Virtual Grasping Using a Data Glove By: Rachel Smith Supervised By: Dr. Kay Robbins 3/25/2005 University of Texas at San Antonio
Motivation Navigation in 3D worlds is awkward using traditional mouse Direct manipulation maybe more intuitive and efficient Traditional immersive environments such as a cave are expensive and inconvenient Gaming industry has brought down cost of devices such as head mounted displays and data gloves 2
Basic Question Is it feasible to use a data glove to manipulate 3D objects in the user s normal desktop environment? 3
Overview Technology Methodology User studies Feature extraction ART2 NN Application Concluding remarks 4
What is a Data Glove? Data gloves measure hand gestures by sensing the bending of finger joints and finger separation. 5
5DT Data Glove 14 Ultra Developed by Fifth Dimension Technologies and is made of a lightweight stretch lycra material. 14 bend sensors use the fiber optic based flexor technologies Emits a stream of raw sensor values 6
What is a Flock of Birds? Flock of Birds (FOB) measures motion tracking of position and orientation of a sensor by a transmitter. 7
Flock of Birds Developed by Ascension Technologies and is a six degrees-of-freedom measuring device. Bird unit Contains its own independent computer Transmitter Transmits a pulsed DC magnetic field that is measured by the sensor Sensor Computes its position and orientation 8
Methodology Perform user studies to get training data Perform feature extraction on training data Train a neural network to recognize grasping objects Develop a simple application to test effectiveness of grasp and motion system 9
User Studies Take a variety of users through a choreographed set of motions which can then be used in supervised learning. Use a neural network as the learning mechanism because of the great variability among users. 10
User Study Architecture File System sampling information Database user information Data Acquisition data Server data Client data 3rd GUI commands C/C++ Application Java Application sensor values sensor values 80Hz Data Glove Flock of Birds 11
3rd GUI 12
User Trial Interaction User sits in front of computer wearing a data glove and FOB sensor while interacting with 3rd GUI Performs different types of hand and arm motions as directed Varying speeds Different positions and orientations Arbitrary motions Each trial duration is approximately 1 minute 13
Trial Types Trial Type 1: Collect grasping information Gestures with the glove Open/close hand Trial Type 2: Collect tracking information Trace object on screen Map physical location of users hand to a position on screen 14
Feature Extraction Select meaningful features that are lowdimensional and capture the underlying coordination of manipulative hand movements. Miriam Zacksenhouse & Paul Marcovici, 2001 15
Feature Extraction (cont.) Manipulative hand movements Coordinated movements of the fingers to manipulate an object within the hand Classified as simultaneous or sequential movements Simultaneous hand movements Simple: participating digits flex and extend concurrently Reciprocal: different digits flex while others extend Coordination provides an inherent representation of the movement Underlying coordination: in-phase or anti-phase Projection of coordination in joint space results in straight-line trajectories Structure of the coordination is captured by the direction of the straight line 16
Example of time domain Index Joint Thumb Joint sensor values sensor values time time 17
Example of joint-space Index v.s. Index Index v.s. Thumb sensor values sensor values sensor values sensor values 18
Feature Extraction Approach Given the training data, compute a vector for each sensor containing the corresponding sensor values Identify most active sensor/joint Contains the vector of slopes Make corresponding vector of slopes available for classification Reduce amount of information needed for classification 19
Feature Extraction Architecture Database sensor values Feature Extraction MATLAB application most active joint answers ART2 NN recognized gesture 20
Joint Space Trajectory - Pinch 21
Adaptive Resonance Theory (ART) Used in pattern-classification problems Solves the stability-plasticity dilemma Instabilities arise from irrelevant input Adapt to novel input Feedback mechanism Resonant state Has previously learned to recognize an input pattern New pattern is stored for the first time Carpenter & Grossberg, 1987 22
Benefits Learn new categories with single novel input Adapt to changes without forgetting previously learned patterns Supports on-line adaptation Eliminates the need for large training data sets and off-line training 23
ART1 versus ART2 ART1 uses binary input ART2 uses analog input 24
ART2 Architecture 25
Offline Training Architecture Database sensor values Feature Extraction MATLAB application most active joint answers ART2 NN recognized gesture 26
Application 3rd Application Take recognized movement and map the hand gesture to a command Grasp, ungrasp, zoom, etc. Take the position and orientation and map the hand motion and speed to the virtual object Translate, rotate, etc. 27
Application Architecture 3rd Application recognized gesture ART2 NN most active joint (X,Y,Z) position Feature Extraction Data acquisition sensor values raw sensor data Flock of Birds raw sensor data Data Glove 28
Conclusion Determine the feasibility of using a data glove for 3D object manipulation in a desktop environment Develop a user study to get training data Perform feature extraction Use the ART2 NN to recognize hand gestures Develop a simple application to test the efficiency of hand gestures and motions 29
References Zacksenhouse M, Marcovici P. Interactive recognition of simultaneous manipulative hand movements. Mechatronics 11 2001; 389-407 Zacksenhouse M, Marcovici P. Inherent structure of manipulative hand movements and its discriminative power. Intelligent Robots and Systems 1 2000; 318-323 Freeman JA, Skapura DM. Neural networks, algorithms, applications, and programming techniques. Reading: Addison-Wesley, 1992. Elliott J.M. and K.J. Connolly. A classification of manipulative hand movements. Developmental Medicine & Child Neurology 26 1984; 283-296 30
Questions 31