Hand Gesture Recognition and Its Application in Robot Control

Transcription

1 IJCSI International Journal of Coputer Science Issues, Volue 13, Issue 1, January Hand Gesture Recognition and Its Application in Robot Control Pei-Guo Wu 1 and Qing-Hu Meng 1 Inforation Engineering College, Henan University of Science and Technology, Luoyang, 47103, China Departent of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, 0085, China Abstract In view of the proble that the accuracy and robustness of hand gesture recognition technology based on vision in the process of using gestures to interact with robots were unstable due to its background, illuination and other factors, this paper presented a gesture segentation and recognition ethod Cobining depth inforation and color iages. First, it used Kinect sensor to obtain depth inforation and color iages, then used depth inforation to pick the hand part fro color iages, and then got gesture iages through color segentation ethod. Second, it calculated HU invariant oents and shape features of the gesture iages as feature inforation. Finally, it used the feature inforation to train the support vector achine, then ipleented hand gesture recognition for static hand gestures. Experiental results show that the ethod has strong robustness to the influence of background interference, illuination variation, translation, rotation and zoo, and can be applied to control intelligent robot. Keywords: Kinect; hand gesture recognition; Hu invariant oents; support vector achine; robot. 1. Introduction With the developent of the robot technology, robots have entered every aspect of huan life. Robots can replace huan to work in the factory, can provide entertainent and help to people in daily life. As a result, interactions between huans and robots are becoing ore and ore frequent. In this case, the traditional huan-coputer interaction technologies based on ouse and keyboard has already could not satisfy people's needs, people are longing for a ore siple, natural and direct way to counicate with the robots. As a coon counication way between people, hand gestures can convey rich and coplete inforation, and play an iportant role in our daily life. The use of hand gestures to control robots can ake people anipulate and interact with the robots easily and conveniently. Therefore, hand gesture recognition is an indispensable key technology in the new generation of huan-coputer interaction technologies [1-3]. Initially, people used data glove to collect the data of hand joints for hand gesture recognition. But the hand gesture recognition Technology based on data glove is not suitable for practical application, because the data glove is expensive and not flexible for user. Now hand gesture recognition Technology based on vision is widely researched and applied, it obtains iages with caera, and used iage processing ethods for processing and analysis, then accoplishes the purpose of hand gesture recognition. Copared with the hand gesture recognition based on data glove, hand gesture recognition based on vision is ore natural and convenient, and has a great application value. In the coplex environent conditions, iages obtained by caera are easily affected by coplex background and illuination variation, that akes it difficult for iage processing and analysis. Therefore, the accuracy of hand gesture recognition is difficult to iprove [4-6]. In this case, this paper presented a hand gesture recognition ethod based on Kinect, it used Kinect to obtain depth inforation and color iages at the sae tie. The depth inforation is only deterined by the distance between object and Kinect, it has strong robustness to the influence of coplex background. The ethod cobined depth inforation with color iages for iage segentation, then extracted feature inforation of the hand iages after segentation, and used the feature inforation to train support vector achine to get classifier, and then used the classifier to recognize hand gestures. Finally, the ethod was used in robot controlling to achieve natural and convenient huan-coputer interaction.. Hand Gesture Segentation Kinect is a 3d sensor which is released by Microsoft Corporation. On the Kinect, there is a color (RGB) caera, an infrared (IR) projector, and an infrared (IR) sensor. With the RGB caera, color iages of the scene in front of the Kinect can be obtained iediately. The IR projector can eit a grid of near infrared (NIR) light in doi: /ijcsi International Journal of Coputer Science Issues

2 IJCSI International Journal of Coputer Science Issues, Volue 13, Issue 1, January front of it. When the NIR light irradiates objects with rough surfaces, according to the distance of the objects, there will be soe different patterns on the surface of the objects. The IR sensor is a CMOS caera that can receive the patterns. Then the patterns are decoded in the Kinect to deterine the depth inforation [7-8]. 11 threshold were replaced with the value of 1, the other pixels were replaced with the value of 0. Finally, binary iages were achieved after the color segentation of hand iages..1 Foreground Extraction In this paper, the frae rate of Kinect was set to 30 Hz, and the resolution of the iage was set to pixels. With the Kinect, color iages and depth inforation could be obtained at the sae tie. It is the value of distance fro Kinect to the objects that is stored in each frae of depth inforation. Then the foreground in the range of 0.5 eters to 1. eters was retrieved fro color iages based on depth inforation. In this way, the iages of hand without any other part of the experienter's body could be obtained; see Fig. 1. H histogra S histogra Fig. Hand color histogras. After skin color segentation, there was soe noise in the binary iages. As a result, edian filtering was necessary before extracting features. Median filter is a kind of nonlinear spatial filters, it provides excellent noisereduction capabilities for certain types of rando noise, with considerably less blurring than linear soothing filters of siilar size. In order to perfor edian filtering at a point in an iage, the values of the pixel in question and its neighbors ust be sorted at first, then deterine their edian, and assign this value to that pixel. Fig. 3 shows the result of edian filtering. Fig. 1 Foreground extraction.. Skin Color Segentation The hand iages which were obtain fro Kinect is a RGB (Red, Green, Blue) color iage. In RGB color space, the color of each pixel is defined by R, G and B values. The R, G and B values have a high correlation, and are easily affected by illuination variation. As a result, it is not suitable for color segentation in RGB color space. Therefore, the hand iages were transfored fro RGB color space to HSV color space. The definition of color in HSV color space accords with the character of huan vision perception very uch. In HSV (Hue, Saturation, Value) color space, H and S values are used to describe the different colors; V value reflects the light and dark of different colors because of light intensity. Hence, color segentation based on HSV color space can eliinate the influence of light intensity [9]. Firstly, the hand iages in RGB color space were transfored to HSV color space. Then, the color histogras of hand were extracted; see Fig.. After analysis of the color histogras, the threshold was deterined as follow: 0 H 15 or 170 H 180, and 50 S 130. The values of pixels within the range of doi: /ijcsi Fig. 3 Median filtering. 3. Feature Extraction Fig. 4 shows the binary iages of hand gestures which are defined in Sebastien Marcel Static Hand Posture Database that were obtained through Kinect. Then the HU invariant oents and copactness of the iages were calculated as feature inforation for hand gesture recognition. 3.1 Hu Invariant Moents In iage processing, oents are usually used to describe geoetrical features of iages, and are used as the basis of classification. If the resolution of the iage is n and 016 International Journal of Coputer Science Issues

3 IJCSI International Journal of Coputer Science Issues, Volue 13, Issue 1, January A B C Five Point V Fig. 4 Binary iages of hand gestures. F ( x, y) is the value at point ( x, y), then the (p + q) th oents and the (p + q) th central oents can be calculated as follows: n x1 y1 n x x1 y1 (, y 0 0 p q y F( x, y), q 0,1,,... p (1) p q ( x x0 ) ( y y0) F( x, y) () Where x ) is the center of gravity in iage, 10 x0, y0. 00 Moents and central oents can be used as shape feature inforation for iages classification, but they are not invariant of transforation, rotation and scale at the sae tie. In 196, HU M.K. put forward the concept of oent invariant, also called Hu invariant oents. Hu invariant oents are invariant of translation, rotation and zoo, therefore they are widely used in iage classification and recognition [10]. Hu invariant oents are constructed with noralized second central oents and third central oents, the noralized central oents can be calculated by zero oent as the following equation: 00 (3) HU invariant oents are calculated as follows: (4) ( (5) 0 0) ( ) (3 1 03) 4 ( 30 1) ( 1 03) (6) (7) 5 ( )( 30 1)[( ) 3( ) ] (3 )( )[ ( 30 1) ( 1 03) ] 6 ( 0 0)[( 30 1) ( 1 03) ] 4 11( 30 1)( 1 03) 7 ( )( 30 1)[( 30 1) 3( 1 03) ] (3 1 30)( 1 03)[3( 30 1) ( 1 03) ] (8) (9) (10) 3. Copactness The sae hand gestures in different iages were not identical because of gesture polyorphis and the influence of the caera angle. Therefore, the Hu invariant oents of iages changed slightly. As a result, the accuracy of hand gesture recognition was reduced. In order to raise the accuracy, two shape features based on copactness were introduced [11-1], they are calculated as follows: l c1 (11) S c 4 S (1) S rect Where l is the circuference of gesture contour, S is the area of gesture, rectangle. S is the area of iniu enclosing rect 4. Hand Gesture Recognition SVM was served as classifier for hand gesture recognition. The first three paraeters in Hu invariant oents and the two shape features were used as feature inforation to train SVM, and then the classifier is obtained. SVM is first put forward by Cortes and Vapnik in 1995, and has becoe a new ethod of achine learning. SVM is based on VC diension theory and structure risk iniization principle, it excels in addressing highdiension and solving sall saple size proble. SVM perfors classification and recognition by finding optial hyperplanes in the feature space. When saples are linear separable, the optial hyperplanes can be find directly. For the linear non-separable probles, it can be solved by introducing slack variables. In fact, ost of the questions doi: /ijcsi International Journal of Coputer Science Issues

4 IJCSI International Journal of Coputer Science Issues, Volue 13, Issue 1, January are nonlinear, they needs to be transfored into linear probles in a high diension space by kernel functions to solve [13-14], three coon kernel functions are as follows: 1. Polynoial kernel function: T d K ( u, v) ( u v c) (13). RBF kernel function: K( u, v) exp( u v ) (14) 3. Sigoid kernel function: T K( u, v) tanh( u v c) (15) RBF kernel function was used in the ethod, it has good stability and wide convergence doain, and has a wider application range than the others. In the experient, in addition to the paraeters in kernel function, the penalty paraeter C is equally iportant. The value of C deterines how iportant the off-group points data is. The bigger the value of C is, the ore iportant the off-group points data is. 5. Results and Analysis 5.1 Experiental Research The hardware equipent of hand gesture recognition includes coputer and Kinect. The operating syste of coputer is Windows 7; the driver software of Kinect is Kinect for Windows SDK v1.8. The software developing platfor is Visual Studio 010. Both Open Source Coputer Vision Library (OpenCV) and LIBSVM tools are used for software developent. With the Kinect, one hundred and sixty iages were obtained randoly for every hand gesture in Sebastien Marcel Static Hand Posture Database. One hundred of the iages were used as training saples, the others were used as test saples. The training saples were used to extract feature inforation, and the feature inforation were used for paraeters optiization with "grid.by" which is included in the LIBSVM tools. Fig. 5 shows the result of paraeters optiization. Then the optial paraeters and feature inforation were used to train SVM. The test saples were used for hand gesture recognition after feature extracting. Table 1 shows the results of hand gesture recognition, and the suggested ethod is ore accurate than the ethod which only use Hu invariant oents as feature inforation. Table 1: Results of hand gesture recognition Recognition Rate(%) Our Method Hu Method Hand Gesture A B C Five Point V Average Robustness Analysis In the coplex environent conditions, coplex background and illuination variation ake it difficult for hand gesture segentation. As a result, the accuracy of hand gesture recognition is reduced. In Fig. 6, a and d are the hand gesture segentation in di light, b and e are the hand gesture segentation in bright light, c and f are the hand gesture segentation in coplex background. Fro Fig. 6, the hand gesture segentation ethod is stable to the influence of illuination change and coplex background, and has strong robustness. In the experient, every hand iage which was used to do hand gesture a b c Fig. 5 Paraeters optiization. d e f Fig. 6 Hand gesture segentation in different condition. doi: /ijcsi International Journal of Coputer Science Issues

5 IJCSI International Journal of Coputer Science Issues, Volue 13, Issue 1, January recognition is different, they have soe changes of translation, rotation and zoo. Cobined with the table 1, we can see that the hand gesture recognition ethod has strong robustness to translation, rotation and zoo. 5.3 Robot Control NAO was used in this experient, it is a prograable huanoid robot developed by Aldebaran Robotics, a French robotics copany. NAO has 5 degrees of freedo, and has any kind of sensors, it is widely used in scientific research and faily service. In the robot control syste, hand gestures (a, b, c, five, point, v) were defined as control coands (go, turn left, turn right, sit down, stand up, say hello) to the robot. The syste used Kinect to obtain hand iages, then do hand gesture recognition. According to the results of hand gesture recognition, the defined control coands were executed. Fig. 7 shows the work flow of robot control syste. The results show that the syste can be used to control the otion of the robot. Kinect Obtain Iages Control Robot 6. Conclusions Fig. 7 Work flow of robot control syste. In this paper, we present a hand gesture recognition ethod based on Kinect. The ethod uses Kinect to obtain hand iages, then perfors skin color segentation and feature extraction to the iages, and uses SVM as classifier. Experiental results show that the ethod has strong robustness to the influence of background interference, illuination variation, translation, rotation and zoo, and can be used to control robot. Future work will focus on the research of dynaic gesture recognition and its application. References Hand Gesture Segentation Execute Control Coand Feature Extraction Hand Gesture Recognition [1] H. Hasan, S. Abdul-Karee, "Huan-coputer interaction using vision-based hand gesture recognition systes: a survey", Neural Coputing and Applications, Vol. 5, No., 014, pp [] Y. Y. Pang, N. A. Isail, and P. L. S. Gilbert, "A Real Tie Vision-Based Hand Gesture Interaction", in the 4th Asia Modelling Syposiu (AMS010), 010, pp [3] M. Takahashi, M. Fujii, M. Naeura, et al., "Huan gesture recognition syste for TV viewing using tie-of-flight caera", Multiedia tools and applications, Vol. 6, No. 3, 013, pp [4] REN Hai-bing, ZHU Yuan-xin, and XU Guang-you, et al., "Vision-Based Recognition of Hand Gestures: A Survey", Acta Electronica Sinica, Vol. 8, No., 000, pp [5] WU Yu, "Gesture Recognition Siulation in Robot Vision Counication", Coputer Siulation, Vol. 3, No., 015, pp [6] Y. Han, "A low-cost visual otion data glove as an input device to interpret huan hand gestures", IEEE Transactions on Consuer Electronics, Vol. 56, No., 010, pp [7] YU tao, Kinect Developent and Application in Actual Cobat, Beijing: China Machine Press, 013. [8] MENG Shang, GAO Chenqiang, and YANG Luyu, "Method of Gesture Recognition Based on Depth Iage", Digital Counication, Vol. 41, No., 014, pp. -6. [9] P. Kakuanu, S. Makrogiannis, and N. Bourbakis, "A survey of skin-color odeling and detection ethods", Pattern Recognition, Vol. 40, No. 3, 007, pp [10] LUO Yuan, XIE Yu, and ZHANG Yi, "Design and Ipleentation of a Gesture-Driven Syste for Intelligent Wheelchairs Based on the Kinect Sensor", Robot, Vol. 34, No. 1, 01, pp , 119. [11] A. Jinda-apiraksa, W. Pongstiensak, and T. Kondo, "A siple shape-based approach to hand gesture recognition", in the 010 International Conference on Electrical Engineering/Electronics Coputer Telecounications and Inforation Technology (ECTI-CON), 010, pp [1] CHEN tao, "Research of Real-Tie Gesture Recognition Technology Based on Depth and Color Inforation ", M.S. thesis, School of Coputer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China, 013. [13] F. Bellakhdhar, K. Loukil, and M. Abid, "Face recognition approach using Gabor Wavelets, PCA and SVM", International Journal of Coputer Science Issues, Vol. 10, No., 013, pp [14] DING Shi-fei, QI Bing-juan, and TAN Hong-yan, "An Overview on Theory and Algorith of Support Vector Machines", Journal of University of Electronic Science and Technology of China, Vol. 40, No. 1, 011, pp Pei-Guo Wu ale, born in 1991, is a aster candidate in coputer application technology at the Inforation Engineering College, Henan University of Science and Technology, Luoyang, China. His research interests include iage processing and recognition. Qing-Hu Meng ale, born in 196, received the Master s degree fro Beijing Institute of Technology, Beijing, China, in 1988, and the Ph.D. degree in electrical and coputer engineering fro the University of Victoria, BC, Canada, in 199. He was a Professor in the Departent of Electrical and Coputer Engineering at the University of Alberta, Canada, fro April 1994 to August 004. Currently, he is a Professor with the Departent of Electronic Engineering, Chinese University of Hong Kong. His research interests are in the areas of bioedical engineering, edical and surgical robotics, active capsule endoscopy, edical doi: /ijcsi International Journal of Coputer Science Issues

6 IJCSI International Journal of Coputer Science Issues, Volue 13, Issue 1, January iage-based autoatic diagnosis, interactive teleedicine and telehealthcare, biosensors and ultisensor data fusion, bio-mems with edical applications, bioedical devices and robotic assistive technologies and prosthetics, adaptive and intelligent systes, and related edical and industrial applications. doi: /ijcsi International Journal of Coputer Science Issues