Visual Interpretation of Hand Gestures as a Practical Interface Modality

Transcription

1 Visual Interpretation of Hand Gestures as a Practical Interface Modality Frederik C. M. Kjeldsen Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 1997

2 1997 Frederik C. M. Kjeldsen All Rights Reserved

3 Abstract This dissertation describes a user interface in which many tasks traditionally performed by a mouse are instead performed using visual recognition of hand gestures. The goals are to explore both how a vision system should be designed to recognize hand gestures, and how they are best used in a general purpose interface. Observed by a camera below the screen, the user manipulates objects directly with gestures incorporating both motion and pose. Task and domain knowledge provide context, allowing real-time recognition on standard PC hardware. A color-based algorithm is trained to segment user's hands from complex backgrounds without visual aids. Training uses a novel combination of both positive and negative data to improve segmentation quality. The apparent path of the hand is smoothed with an algorithm which reduces the types of noise inherent in the domain but leaves a cursor motion on the screen that feels natural for the user. Salient features of the motion are extracted, including a newly discovered natural gesture (a Comma ), which helps provide punctuation for each gestural sentence. Neural networks are trained to classify the pose of the user's hand from cropped and preprocessed images. The nets correctly classify 90-95% of the hand images in real time. A transition network encodes the interaction language. It controls the application of feature extraction operators and interprets their results to determine when to perform actions on the user's behalf. The style of interaction is based on studies of natural gesticulation and incorporates various features designed to make it natural and easy for the user to remember. The system demonstrates a 80-90% success rate on most tasks. Object selection time for large objects is demonstrated to be equal or superior to that of a mouse. Object selection performance is modeled accurately by augmenting Fitts' Law with terms for lag and random cursor noise. Finally, the suitability of gesture for this type of task is considered. Various interaction styles are examined, and problems specific to hand gesture are discussed.

4 Acknowledgments I would like to express my thanks to IBM for the generous support of this work, via the Resident Study Program. Several individuals deserve special mention. My advisor, John Kender, has given his support in many ways. Ross Bevridge's suggestions helped shape the direction of this work, and Steve Feiner's comments helped to make the thesis much more complete. Several members of the T.J. Watson community have been very helpful, both as colleagues and laboratory rats. In particular, thanks to Jon Connell for both inspiration and his many excellent comments, as well as to Sharatchandra Pankanti, Michael Yao, Chitra Dorai, and Lisa Brown. Finally, apologies to my son, Joseph, for having to do without his father so much during the precious first year of life, and sincere thanks to his mother, Lorraine, both for picking up the slack when I was not there, and suffering my moods when I was. i

5 Contents Chapter 1: Introduction Why Gesture How should it be used? Why Vision Scope of problem Difficulties Overview of Thesis... 9 Chapter 2: Background Hand Gesture Theory Hand Gesture Recognition Hand Segmentation Pose Recognition Motion Interpretation Applications of gesture recognition Virtual Environments Gesture in Traditional Interfaces...28 Chapter 3: System Description Overview Design Discussion Hand Segmentation Overview Color Predicate and Training Segmentation Process Design Discussion Hand Tracking Design Discussion Motion Smoothing the Hand Path...50 ii

6 3.4.2 Extraction Motion Features Design Discussion Pose Recognition Gesture Interpretation Design Discussion Implementation Details and Parameters Hardware Segmentation Tracking Motion Pose Recognition Interaction Language Details Window System Interface...94 Chapter 4: Performance Evaluation Segmentation Overall Performance Calibration Performance in Different Environmental Conditions Performance on Different Skin Tones Non-Hand Skin Regions Other Issues Affecting Segmentation Quality Hand Motion Tracking Smoothing Algorithm Performance Object Selection Performance Subjective Evaluation of Tracking Performance Pose Recognition Evaluating Network Performance Network Training Sources of Error Network Weight Analysis Variations iii

7 4.4 The System as a Whole Speed Task Performance User Comments Chapter 5: Discussion Vision Systems for Hand Gesture Recognition Segmentation Tracking Motion Feature Extraction Pose Recognition Language Representation General Considerations Hand Gestures as an Interface Modality Characteristics of Free-Hand Gesture Designing an Interface for Hand Gestures Climbing the Learning Curve Design of a Practical Gesture System Chapter 6 Summary and Conclusions Summary In Conclusion References 190 Appendix 196 iv

8 List of Figures and Tables Chapter Physical layout System's view of user...30 Image labeled by CP and the largest connected component Weighting function around CP training data...35 User training system...35 Segmentations of Figure 2 for tracking and pose recognition...36 Optimal CP and CP produced by histogramming the positive training examples Color Predicates trained using simple update and with Gaussian smoothing and subtraction Images segmented using the CPs in Figure Segmented images of the user pointing to the four corners of the screen...44 The centroid of the user's hand pointing to the corners of the screen forms a quadrilateral in image space...45 The centroid of the hand as it follows a grid in screen space, forms a warped grid in image space Sigmoid force scaling functions...52 The hand backing up behind the cursor between cycles and causing overshoot in a simple smoothing algorithm Force applied to the cursor versus hand displacement...55 Table: Motion Features Pose recognition network architecture Various appearances the Point pose takes on...66 Hand pointing to the top and bottom of the screen Color to gray conversion Two poses very similar in joint angle space, but easy to differentiate in image space Two pointing poses with corrupted outlines A pointing pose with the finger removed and a fist pose...71 Two extremes of roll in a pointing pose...74 Transition network for window control task...76 Table: The actions which can be performed at each node...77 Interaction language using only motion features...80 Three CP training templates...89 v

9 Chapter Segmentation performance, the good, the average, and the ugly...97 Point missing a finger...97 Fist with hole...97 Example of the face and arm extracted with the hand Example hand images from the PCN training set Hand location before and after smoothing Table: Selection times for 1 inch target with free-hand pointing Time to select a screen object versus its size in inches Table: Selection time in seconds versus target size in inches Predicted and actual mouse selection time for objects of various sizes Table: Probability of cursor landing in target in any one cycle for various cursor error distributions and target sizes Table: Expected number of cycles it will take for the cursor to land inside the target for 3.5 consecutive cycles at various levels of noise Predicted and actual selection time for targets of various sizes using free-hand pointing Predicted selection time from simply increasing tracking rate Predicted free-hand selection time with a reduced level of random noise and with no noise Selection time performance for realistic targets of tracking rate and noise Predicted free-hand selection times under ideal conditions Examples of the three pose classes differentiated by one of the PCNs Total classification performance versus training cycle for the training and test sets Weights in a typical pose classification network Example images for network weights discussion Total classification performance during training for binary pose images Classification performance for palm poses during training for binary pose images Table: Results of system task testing Table: Percentage of total errors by category Chapter Alternate interaction language for the window control task, using the pose of the hand and the motion that occurs after it to signal an action Interaction language allowing multiple actions, separated by a comma Menu layout better suited for hand gesture vi