GESTURE RECOGNITION FRAMEWORK FOR MULTIMEDIA CONTENT VIEWER CONTROLLING

Transcription

1 GESTURE RECOGNITION FRAMEWORK FOR MULTIMEDIA CONTENT VIEWER CONTROLLING Michal Lech, Bozena Kostek, Andrzej Czyzewski, Piotr Odya Gdansk University of Technology, Multimedia Systems Dept., Gdansk, Poland {mlech, bozenka, piotrod, Abstract: In the paper a system for controlling a multimedia content viewer by hand gestures is presented. First, selected methods used for gesture recognition are described. Two different application cases of the system, i.e. for multimedia presentation purposes and for multimedia content viewing are outlined. Moreover, a proposal of improvement of the system combining these approaches is also given. The system work cycle is reviewed. The results of the system tests are provided. 1 Introduction In the future probably no one will be any more confused spotting in a public building in front of him a monitor screen presenting information requiring interaction to be handled properly, while no buttons or computer mouse are available. As an example of such devices touch screen viewers in photo laboratories, customers can use to choose photos to be printed or touch screen information points in department stores can be mentioned. An advantage of using such a way of human-computer interaction in such places over using computer mouse is undeniable but frequent screen cleaning remains indispensable. In the paper a system overcoming this drawback, thanks to its touch-less operating mode, is presented. 2 Gesture recognition methods Gesture recognition is a complex problem requiring mathematical modeling and artificial intelligence methods. The mathematical modeling methods can be divided into two groups depending on the selecting static or dynamic gesture handling. In the static mode different shapes formed of palm and fingers positioning are to be recognized, while in the dynamic gestures mode, hand movements interpreting is the problem to be solved. Among the first group methods as an example one can mention edge direction histograms [1, 9], elastic graphs [11, 20, 21], active contours [13, 14], or deformable templates [17, 22]. All these methods are used for palm Research funded within the project No. POIG /08, entitled "Elaboration of a series of multimodal interfaces and their implementation to educational, medical, security and industrial applications". The project is subsidized by the European regional development fund of the and by the Polish State budget. mathematical modeling. As the methods of the second group Hidden Markov Models [6], Kalman filters [2, 12, 15, 16, 18], Volterra filters [3, 8], Dynamic Time Warping [5, 7, 19] or molecular filters can be mentioned. The multimedia content viewer controller described in the paper bases so far on the dynamic gestures, thus herein only some examples of methods of the second group provide the subject of review. Typical methods of dynamic systems modeling are probabilistic models based on conditional probabilities utilizing discrete state values. As a common example Hidden Markov Models can be mentioned. The model is described by probabilities distribution of state transitions, probabilities distribution of observations in states and initial state distributions. Information about the state is not known (is hidden). Only the probability function assigned to the state is given. For each dynamic gesture a separate Hidden Markov Model is created. Gesture recognition is based on estimating the probability with which the observed sequence can be represented by a particular model [6]. With designing the system of dynamic gesture recognition the problem of effective object (e.g. hand) tracking is associated. This task can be performed using Kalman filters. The Kalman filter enables to determine the most probable value of the particular parameter of an object, such as velocity or position, considering a number of observations of tracked object. The assumptions are: the state of an object in time t i+1 is linearly associated with state in time t i and the First Order Markov criterion is met. These constraints are not put on the method of molecular filters, in which probabilities distributions are approximated using clouds of weighted particles [2, 12, 15, 16, 18]. 3 Multimedia content viewer controller The presented technology of controlling multimedia content viewer by gestures is based on following equipment: multimedia projector, two USB cameras, screen for projected image, standard PC (2GB RAM, dual core processor). The projector is hung under ceiling and one of the cameras is mounted directly below or on the projector with whole area of the screen in the lens field of vision. The second camera is placed near left or right side

2 of the screen angled 90 degrees with the ceiling camera screen line. Such a system provides the possibility of using it not only as a tool for individual multimedia viewing but also as a tool for presentations, in which the whole area of the screen should be visible to the audience and not occulted by the user. A software for such a system runs as a background thread able to control typical multimedia content viewers such as IrfanView, ACDSee, Picasa, Microsoft image viewer and Microsoft PowerPoint. When using a system only as a tool for individual multimedia viewing, the user is situated on the line designated by a projector and a screen (Fig. 1). Controlling is based in this case on the recognition of basic hand movements, i.e. moving hand from the middle to the left side for displaying previous image, to the right for displaying next image, up for zooming in and down for zooming out. Fig. 1. A way of using the engineered system for individual multimedia content viewing For presentation purposes the user stands near the side of the screen opposite to the side where the camera is placed and makes gestures towards camera lens (Fig. 2). Controlling is based on recognizing the same set of hand movements. Moving hand from the middle to the left side executes showing previous slide and to the right side next slide. Moving hands up or down has no assigned operation. Additionally changing position of a hand cause changing mouse cursor position. Fig. 2. A way of using the engineered system for presentations The way of proceeding when using in presentation can be regarded as somewhat non-intuitive. In order to indicate something on the screen the user has to remember not to direct his hand at the screen but toward the camera objective, instead. For this reason an improvement based on a simultaneous analysis of video stream from both cameras is being considered. Combining two sheer to one another two-dimensional images makes it possible to retrieve information about the position of a hand in threedimensional space in order to perform its appropriate translation for position on the screen. In such a case in order to indicate something on the screen the user aims his hand directly at the screen. The crucial problem connected with presented solution is specifying appropriate relationship between distance from the user to the cameras, camera resolutions, and angle between straighten arm and the screen plane. On the relationship of these parameters the final system accuracy depends. 4 Implementation and tests The interface is being implemented in the Java language. The choice of using this particular language and platform was due to the presence of many various image capturing and processing libraries on LGPL license, which can be easily adopted to the specific environment. In Fig. 4 a block diagram of a prototype of the gesture recognition system is presented. Analyzing Fig. 4 one can notice that the system work cycle has been divided into three s. The first one regards calibration operations, both manual, performed by user and automatic, based on user s indications. The main aim of this is to determine the effective area of an image grabbed from the camera mounted on the ceiling projector. This area, regarded as the view of desktop and not the surroundings, is determined by the user pointing positions of desktop corners in the frame (Fig. 3).

3 Fig. 3. Determining effective area of an image grabbed from the camera placed under the projector Basing on these positions, scaling of the desktop image is performed in such a way, that the image and the image grabbed from the camera have identical dimensions. Also, basing on these values perspective correction is performed. Optionally, camera calibration aiming at setting parameters in such a way that images grabbed from the camera and desktop images are similar in the sense of brightness, contrast and sharpness, is performed. For the same reason optional image processing can be performed during which optimal parameters, used in gesture recognition, are calculated. The second is responsible for gesture detection. The image of the desktop and the view of the desktop retrieved from the camera are subtracted one from another. In the current version of the prototype four gestures, i.e. hand motions in left, right, up and down direction are recognizable. The recognition of position of a hand in the frame is based on calculating pixel similarities between desktop image and the frame obtained from a camera with given similarity threshold. Parameter determining minimum number of adjacent pixels, formed in cross shape, stating about recognizing object as a fragment of a hand and not a background is given as an algorithm input. Collecting hand positions enables to determine the gesture made, while gathering information about sequence of simple gestures gives possibility of creating and recognizing complex gestures. In the last, called executive, an operation assigned to particular gesture is performed. The system has been tested in the image viewing mode. The reliability, regarded as the ratio between properly recognized gestures and all gestures was rated 90%. Worth noticing is the fact that improper recognition was mainly the effect of performing two gestures, one instantly after another. In fact, after recognizing a gesture, 2 second pause during which no actions are recognized is introduced in the system. Such a pause is a temporary solution for a problem of lack of perfect synchronization between processes of grabbing images from the camera and retrieving corresponding desktop images. If providing appropriate algorithm for the synchronization or using the system regarding this 2 second pause the reliability shall be even higher. During the tests, resolution of 320x200 pixels was used for examining hand gestures recognition. Such a resolution was sufficient for recognizing simple gestures, i.e. moving hand in various directions and provided reasonable time efficiency. However, for accurate hand positions tracking, e.g. for handwriting recognition purposes, resolution of at least 640x480 pixels should be used. It was stated that such a resolution introduces latency making the system highly impractical. The delay was mainly associated with the necessity of performing perspective correction in each frame. As a solution one can consider preliminary image scaling to obtain the effective area only, then performing the perspective correction and finally trimming. Another problems observed during the tests were improper image scaling when using wide screen monitor and 4:3 projector, and high frame rate dependency on light conditions.

4 Displaying calibration image Grabbing frame from a camera Grabbing desktop image Calibration Perspective correction Effective area determination Camera calibration processing Grabbing frame from camera Grabbing desktop image Perspective correction processing subtraction Gesture detection Hand position detection Adding to hand positions collection Gesture recognition Adding to gestures collection Running operation Executive Fig. 4. Block diagram of a gesture recognition system 5 Conclusions The research on computer gesture recognition problem shows that creating effective human-computer interaction interface is indeed a complex task. Basic methods used currently in the prototype of the gesture controlled computer system enable to recognize simple gestures, such as hand motion in left or right direction, especially in specific circumstances ensuring reliable results. For a reliable gesture controlling some more sophisticated methods, such as Hidden Markov Models, Kalman filters, Volterra filters should be used. On the other hand taking advantages of too many sophisticated methods in one system can lead to unacceptable latency problems making the interface ineffective and difficult to control. Analyzing specificity of using gestures as a mean to control a computer and considering possibilities created thanks to this particular way of human-computer interaction method one can notice that world widespread method of working with computer basing on window interface designed according to mouse handling is not necessarily the most ergonomic one. Therefore, developing an effective and reliable technology of the computer controlling using gestures should include designing a new, from a point of view of a user, kind of interface, eliminating the presence of a cursor on a screen. In the further stage of development, to achieve more reliable gesture recognition, basic methods currently used in the prototype will be extended with Kalman filtering for hand position tracking.

5 REFERENCES [1] Birk, J.; Kelley, R.; Chen, N.; Wilson, L, Feature Extraction Using Diameter-Limited Gradient Direction Histograms, Pattern Analysis and Machine Intelligence, IEEE Transactions on, Vol. PAMI-1, Issue 2, pp , April 1979 [2] Clipp B., Welch G., Frahm J., Pollefeys M., Structure From Motion via a Two-Stage Pipeline of Extended Kalman Filters, Proceedings of the British Machine Vision Conference (BMVC 2007), September [3] Dayong Zhou, DeBrunner, V.E., Efficient adaptive Volterra filters for active nonlinear noise control with a linear secondary-path, Circuits and Systems, ISCAS '04. Proceedings of the 2004 International Symposium on, vol. 3, pp. III , May 2004 [4] Da-Rui Sun; Le-Nan Wu, A local-to-holistic face recognition approach using elastic graph matching, Machine Learning and Cybernetics, Proceedings International Conference on, vol. 1, pp , 4-5 Nov [5] Giorgino, T.; Tormene, P.; Quaglini, S., A Multivariate Time- Warping Based Classifier for Gesture Recognition with Wearable Strain Sensors, Engineering in Medicine and Biology Society, EMBS th Annual International Conference of the IEEE, 22- pp , 26 Aug [6] Morguet, P.; Lang, M., Spotting dynamic hand gestures in video image sequences using hidden Markov models, Processing, ICIP 98. Proceedings International Conference on, vol. 3, pp , 4-7 Oct [7] Muscillo, R.; Conforto, S.; Schmid, M.; Caselli, P.; D'Alessio, T., Classification of Motor Activities through Derivative Dynamic Time Warping applied on Accelerometer Data, Engineering in Medicine and Biology Society, EMBS th Annual International Conference of the IEEE, pp , Aug [8] Kenji Nakayama, Akihiro Hirano, Hiroaki Kashimoto, A Lattice Predictor Based Adaptive Volterra Filter And a Synchronized Learning Algorithm, XII European Signal Processing Conference, September 6-10, 2004, Vienna [9] Pinheiro, A.M.G., Description Using Scale-Space Edge Pixel Directions Histogram, Semantic Media Adaptation and Personalization, Second International Workshop on, pp , Dec [10] Qiuyu Zhang; Fan Chen; Xinwen Liu, Hand Gesture Detection and Segmentation Based on Difference Background with Complex Background, Embedded Software and Systems, ICESS '08. International Conference on, pp , July 2008 [11] Stamou, G.N.; Nikolaidis; Pitas, I., Object tracking based on morphological elastic graph matching, Processing, ICIP IEEE International Conference on, vol. 1, pp. I , Sept [12] St-Pierre, M.; Gingras, D, Comparison between the unscented Kalman filter and the extended Kalman filter for the position estimation module of an integrated navigation information system, Intelligent Vehicles Symposium, 2004 IEEE, pp , June 2004 [13] Sundaramoorthi, G.; Jackson, J.D.; Yezzi, A.; Mennucci, A.C, Tracking With Sobolev Active Contours, Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, Volume 1, pp , June 2006 [14] Sundaramoorthi, G.; Yezzi, A.; Mennucci, A.C., Coarse-to-Fine Segmentation and Tracking Using Sobolev Active Contours, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 30, Issue 5, pp , May 2008 [15] Welch G., HISTORY: The Use of the Kalman Filter for Human Motion Tracking in Virtual Reality, Presence: Teleoperators and Virtual Environments, 18(1), 2009 [16] Welch G., and Bishop G., An Introduction to the Kalman Filter, SIGGRAPH 2001 course 8. In Computer Graphics, Annual Conference on Computer Graphics & Interactive Techniques. ACM Press, August [17] Xun Wang; Lei He; Yingjie Tang; Wee, W.G., A divide and conquer deformable contour method with a model based searching algorithm, Systems, Man, and Cybernetics, Part B, IEEE Transactions on, vol. 33, Issue 5, pp , Oct [18] Yang H., Welch G., Model-Based 3D Object Tracking Using an Extended-Extended Kalman Filter and Graphics Rendered Measurements, in Proceedings of 1st Computer Vision for Interactive and Intelligent Environments (CV4IIE) workshop, 2005 [19] Yaniv, R.; Burshtein, D., An enhanced dynamic time warping model for improved estimation of DTW parameters, Speech and Audio Processing, IEEE Transactions on, vol. 11, Issue 3, pp , May 2003 [20] Zafeiriou, S.; Tefas, A.; Pitas, I., Advances in Elastic Graph Matching for Frontal Face Verification, Computational Intelligence in and Signal Processing, CIISP IEEE Symposium on, pp , 1-5 April 2007 [21] Zafeiriou, S.; Tefas, A.; Pitas, I., Exploiting discriminant information in elastic graph matching, Processing, ICIP IEEE International Conference on, vol. 3, pp. III , Sept [22] Zhong Jing; Robert Mariani, Glasses detection and extraction by deformable contour, Pattern Recognition, Proceedings. 15th International Conference on, vol. 2, pp , 3-7 Sept 2000