Human Computer Interaction using Facial gestures to control a Wheelchair for tetraplegic users

Transcription

1 8 al 10 de octubre de 2008, Cuernavaca, Morelos., México. Human Computer Interaction using Facial gestures to control a Wheelchair for tetraplegic users R. P. G. Rubén Posada Gómez, L. H. S. M. Luis Humberto Sánchez Medel Resumen: La Interacción Humano Computadora (HCI) es un campo en constante evolución, teclado, mouse y joystick son algunos de los dispositivos utilizados para interactuar con la computadora, sin embargo nuevas formas de interacción se están desarrollando como el control por audio y/o video. Los pacientes impedidos como los tetrapléjicos necesitan encontrar nuevas formas de interactuar no solo con la computadora sino también con el mundo que los rodea; las sillas de ruedas actuales no son adecuadas para estos usuarios debido a su falta de movimiento en sus brazos y piernas. Los gestos faciales son usados para expresar alegría, tristeza, enojo etcétera, debido a que los usuarios impedidos solo pueden hacer uso de su rostro para expresarse, un HCI puede hacer uso de estas expresiones para controlar la dirección de una silla de ruedas utilizando una webcam para la adquisición de la imagen. Palabras Clave: HCI, Silla de ruedas inteligente, reconocimiento de gestos faciales, visión máquina. Abstract: The Human Computer Interaction (HCI) is a field in constant evolution, keyboard, mouse and joystick are some dispositives used to interact with the computer, however new forms are being developed like audio and/or video control. The handicapped patients such as tetraplegic need to find new ways to interact not only with the computer but also with the world around them; the current wheelchairs aren t suitable for these users due to their lack of movement in their arms and hands. The facial gestures are used to express happiness, sadness, angriness etcetera; due handicapped user can t use but their face to express themselves a HCI can use this expressions to control a device such as a robotic wheelchair. The objective of this article is to develop the results of a HCI system using facial Luis Humberto Sánchez Medel, Instituto Tecnológico de Orizaba, Av. Instituto Tecnológico No. 852, México, lsmedel@gmail.com Rubén Posada Gómez, Instituto Tecnológico de Orizaba, Av. Instituto Tecnológico No. 852, México, ruben.posadagomez@gmail.com gestures for the handicapped user to control the direction of a wheelchair using a webcam for image acquisition. Keywords: Intelligent wheelchair, Facial gestures recognition, machine visión. Introduction The mechanical wheelchair (NWs) is the most common Assistive Technology used to transport the handicapped patient and aged people. The evolution of the NWs has made possible the control of the wheelchair using a joystick in an electric power wheelchair (EPWs) [1] that requires less effort by the user; however the patients with tetraplegic can t use this kind of wheelchairs due their lack of movement in their arms and hands. Recent investigations are focus to develop new forms to transport the user in a safer and comfortable way, when sensors and computer are added to an EPWs an intelligent wheelchair (IWs) is obtained [2]. The IWs can include different kinds of improvements as design or operation including devices that allow to go up the IWs to a provided car of especial harness where the IWs takes the most convenient position to be able to ascend to the vehicle. An array of sensors and cameras are included as much in the automobile as in the IWs [3]. Additionally, environment looking for border of walls [4]. Previous works propose new forms of HCI for handicapped patients to control the IWs by speech [5] or even identifying the form and position of lips [6]; however both systems are susceptible of undesirable movements during a conversation while the seat is operated, or when the IWs is located in noisy environments. Additionally, works have been developed in the detection of user s face for controlling the wheelchair, detecting the iris position with a webcam located in front of the user [7], as the location of iris is very dependant of the camera s position, a user s face control is also proposed [8]. However this type of IWs needs that user sees only in the direction of displacement, and cannot be used by quadriplegic people with problems at neck level. 405 Page 1

2 In this article are presented the results of a new form of HCI using facial gestures to control a wheelchair, the machine vision program makes use of skin detection, face detection algorithms, morphological operations to extract the gestures and match patterns algorithms to classify the facial gesture. The human spirit is unstoppable, the Paralympics competences are a clear example of the will and tenacity that handicapped patients has, so it s important to develop new and better technology to allow them to reintegrate to the world. Methodology The machine vision system is divided in the stages that Fig. 1 shows. The configuration of the system refers to select an available webcam, so there can t be any configuration issue; the software used to control the IWs is LabVIEW v7.0. The acquisition of the image is realized by means of a webcam, because is cheap and available. manually and is processed to obtain its representation in the diverse space color and its average value within the selected area. The equations 1-10 show the correct representation of the RGB colors in the different spaces of color, the equation 11 shows the correct form to obtain the average value of an area of pixels.,,. (1) /, 0, (2). 2 /, 0,2 (3) min (4) 0,, 1/2 (5), 1/2 (6), 0,1 (7) (8) (9) (10),, (11) Fig. 1. Necessary steps to perform a facial gesture recognition. Skin detection The skin detection task is usually the first stage in many applications like face detection or face/hands recognition and has great acceptation due is capable to separate the background of a person [9]. Skin database. The first step is create a skin database of images, so we can train the detector, our skin data base consist of 2341 images that contains skin information in different perspectives, shadows and lighting conditions. Once the skin database is realized, we select manually the types of skin color to detect, the selection must not include moustache, hair and saturated skin color image. The skin detection task is realized in 5 spaces of color: RGB, HSI, HSV, HSL and YCbCr plus the movement. Each area of skin selected The data obtain is a histogram that represents the intensity of the skin-like pixels, an example of the histograms is the Fig. 2. The numeric skin data is postprocessed to improve the skin detector. The criteria of the improvement is based how much a value repeat itself, so the less probable numeric data is eliminated, this reduce the memory consuming and improve the velocity of the skin color detection. Fig. 2. Histograms of skin-like pixels in the H color space. Pag. 2

3 Skin detection by movement. The human body is in constant movement although we are not aware of, so when we compare two images of a sequence we can see some differences, this aids the detector to identify the movement. The equation 12 and 13 allows identifying the differences in two images in sequence. Equation 12 is a morphological filter that eliminates the effect salt & pepper (random noise), the equation 13 realizes the difference between two images in sequence.,,,,,,,, (12),,,, 1 (13) Five spaces Skin color detection plus movement. The color is a perceptive phenomenon so it depends of lighting conditions, this is a problem when the illumination changes constantly. Based on the different spaces of color it s possible to improve a skin color algorithm, the movement represents a sixth space although it s not color, aids to detect the skin. The equation 14 allows detecting the skin using the 6 spaces of information; the resulting image is show in Fig. 3. (15) The filling hole morphological operation is implemented by a dilatation algorithm. Dilation eliminates tiny holes isolated in particles and expands the particle contours according to the template defined by the structuring element. This function has the opposite effect of an erosion because the dilation is equivalent to eroding the background. The dilatation operation is realized by equation 16. The result image is shown in Fig. 3 (segmented), the skin area pixel are subtracted from the original image. (16) Face detection The face detection is performed by taking into account the position of the user that is in front of the webcam and with the face uncovered. The face detection algorithm is based in circle detection, because the face is an oval of skin pixels, so identifying the position of all the circles and selecting the biggest circle provide us the face position. The Fig. 4 shows the complete algorithm to face detection in images. skin RGB HSI HSL HSV YCbCr MOV (14) Fig. 3. The result image after the sixth space is a gray image, where the lighter pixels represent skin-like pixels and darker represent non-like skin pixels Morphological filters. The result image in gray scale is processed to obtain the skin pixels using a threshold operation, Fig. 3 show the binary image obtained, we can appreciate some areas of pixels missing and other lines that aren t pixels. The criteria to select which morphological filter lie in the nature of the image to analyze, we know that skin pixels must be reunited in big areas and does not must be lines. A morphological operation such as remove particle and fill hole operation are required. The remove particle operation remove lonely particles, is realized with an erosion process as defined by equation 15 [10]. Fig. 4. The face detection algorithm uses the information of the skin detection to identify the circle sin the image, once the bigger circle is identified, the face is cropped to reduce the time to process the facial gestures Facial gestures detection using morphological filters. The facial gestures is information contained in the internal face contour as the Fig. 4 shows, once the face is cropped an image differential is used to remove all color and leaving only the edges of the image, the result Pag. 3

4 is gray edge image, that represents the shadows, this image is threshold to perform binary morphological operations. The important information are the eyes (open/closed) and mouth (open/closed), using a remove border filter and remove particle filter leaves the eyes and the mouth as seen in Fig. 4. Differential Image. The differential image eliminates all color tones and focus in the shadows of the image; it s obtained by the absolute difference between two areas of neighborhood pixels from left to right and up to bottom as equation 22 shows. limits only the area where the information must be found, this is the reason to use skin and face detection. The concept of normalized correlation refers to consider a subimage w(x,y) of KxL size in a Image f(x,y) of MxN size where K M and L N, the correlation between w(x,y) in a (i,j) is given by the equation 18 [10]. The correlation process is very sensitive to changes of amplitude in the image, as intensity; although this sensibility can be obvious if the equation is normalized as 19., 1,,, 1 (17) The Fig, 4 shows the result of the differential image (top right and up), now the morphological operations are use to remove the unwanted information as contour of the face, ears, hair, etcetera. The morphological operations to improve the result is the reject border operation; it s an advanced morphological operator built upon the primary operations because uses basic operations as dilation and erosion with conditional combinations. Pattern matching. The pattern matcing technique use a template that represents the objects of interest, then searches for instances of the template in each acquired image, calculating a score for each match. This score relates how closely the template resembles the located matches. This technique can locate templates regardless of lighting variations, noise and geometric transformations such as shifting, rotation or scaling.,,, (18),,,,,,, (19) The pattern matching is based in the normalized correlation that is the most common method to find the model of the image; it s based in series of multiplications so it consumes a considerable time. To improve the speed of the correlation process the size of the image must be reduce so that the search region Fig. 5. The mouth is detected using pattern matching, once is found the eyes are detected simply by adding x, y coordinates, to determinate if an eye is open or closed a histogram is used to measure the intensity of the pixels in the eyes area. The object to find in the image is the open mouth because is a sign that most the people can do; once the mouth is located the eyes are detected simply by adding x, y coordinates. To determinate if an eye is open or closed a histogram is used to measure the level intensity of the pixel in the eyes. The Table 1 shows the classification of the direction based in the facial gestures. The direction is send to an electronic circuit in the IWs that receive the signal and process the information to detect obstacles. Table 1 Classification of the IWs direction using eyes and mouth. Mouth Right eye Left eye Direction BYTE Undetected X X Alto 0h Detected Open Open Adelante 1h Detected Open Closed Derecha 2h Detected Closed Open Izquierda 3h RESULTS. The Fig. 6 shows the result of the skin color detection algorithm, the morphological operations aids to identify skin areas in the image, removing lonely particles and Pag. 4

5 filling the holes of skin pixels areas, the time of the skin detection is 118 ms. The algorithm has been proved in images that are contained in the image data base and outside of it with a webcam. Table 2 Results of the face detection algorithm in diverses lighting conditions Lighting condition TP FP Percentage of right results Average time (ms) DARK 50 4 NORMAL % % 123 SATURAT ED % 138 CONCLUSIONS. Fig. 6. Results of the skin color detection algorithm in images that aren t contained in the database and with different skin colors and lighting conditions The diverse algorithm presented together can be used to perform facial gesture detection; the patter matching is a technique used to located the mouth in the image however if the searching area isn t limited can produce false positives, the skin and face detection limits its searching area improving the detection. Table 3 Results of the facial gestures detection, these results are taken from a webcam video sequence. Lips detection Right eye Open/closed detection Left eye Open/closed detection Average time (ms) TP 94% 82% 84% 148 FP 6% 12% 26% 143 Fig. 7. Results of the face detection algorithm in images with different persons, skin color and perspective The results of the face detection algorithm is shown in Fig. 7, the images are taken from the image database and sequence of images of a webcam; it takes 120 ms to detect the face of the user, so the maximum rate is 8 frames per second. The detection is successful in normal lighting conditions (blank light that not saturates the webcam sensor). The face detection algorithm produces false positives when the webcam has a bad illumination that provoke that different pixel acquire skin-like colors, this is due the color is an perceptive phenomenon. The table 2 shows the result of the face detection algorithm. The results of the facial gesture detection are shown in Table 3 and the program in Labview is shown in Fig, 8. Fig. 8 The facial gesture detection program was developed in LabVIEW v 7.0, the figure shows the interface for the user. [1] AMundson JS, Amundson SG., A joystick controlled wheelchair., Biomed Sci INstrum 1991;27:131-3 [2] T. Gomi and A Griffith, Developing intelligent wheelchairs dor the handicapped Lectures Notes in AI: Assistive Technology and Artificial Intelligence, Springer vol. 1458, pp , 1998 [3] Humberto Semeno-Villata and jhon Spletzer, Vision-based Control of a smart Wheelchair for the Automated transport and retrieval System (ATRS) p.1, Department of computer science and engineering Lehigh University, Pag. 5

6 [4] Trahanias, P. E. et al, 1997 Navigational support for robotic wheelchair platforms: an approach that combines vision and range sensors Proceedings of the 1997 IEEE International Conference on robotics and automation, Albuquerque, NM, 1265, 2005 [5] Pacnik, G., K. Benkic and B. Brecko, Vouce Operated intelligent wheelchair-voic, Proc. Of the IEEE International Symposium on Industrial Electronics, vol 3, pp , 2005 [6]Ulrico Cnazler, Karl-Friedrich Kraiss, Person-Adapative Facial Feature Analysis for Advanced Wheelchair User-interface, Conference on Mechatronics & Robotics 2004, Volume Part III, pp , September 13-15, Aachen, Sascha Eysoldt Verlag. [7] L. M. Bergaze, M. Mazo, C.T. San Juan, J. A. Herradas Guiado de un móvil mediante los movimientos oculares. I Jornadas de Inteligencia Artificial, control y sistemas expertos, E.U. Politecnica Universidad de Alcala, pp [8]P. Jia and H. Hu (2005), Head Gesture Based Control of an Intelligent Wheelchair, p.1, The 11th Annual Conference of the Chinese Automation and Computing Society in the UK (CACSUK05), Sheffield, UK, 10 September [9] Stan Z. la et al, Handbook of face recognition, Springers, ISBN X, 2004 [10] NI VISION, NI VISION: IMAQ VISION CONCEPTS MANUAL, National Instruments., USA Currículo corto de los autores Rubén Posada Gómez Ingeniero Electrónico egresado del Instituto Tecnológico de Orizaba en Obtuvo el grado de Maestría en Ciencias en 2003, Doctorado en 2005 en el Institud Nacional Polytechnique de la Lorraine (inpl), Nancy,Francia Ensem, Cran. Grupo de Ingeniería para la Salud y Postdoctorado en el Centro de Investigación de Matemáticas en 2008 en Guanajuato, GTO. México. Luis Humberto Sánchez Medel Ingeniero Electrónico egresado del Instituto Tecnológico de Veracruz en 2005, Maestro en ciencias en Ingeniería Electrónica Digital en 2008 en Orizaba, Ver. México. Pag. 6