Virtual Grasping Using a Data Glove

Transcription

1 Virtual Grasping Using a Data Glove By: Rachel Smith Supervised By: Dr. Kay Robbins 3/25/2005 University of Texas at San Antonio

2 Motivation Navigation in 3D worlds is awkward using traditional mouse Direct manipulation maybe more intuitive and efficient Traditional immersive environments such as a cave are expensive and inconvenient Gaming industry has brought down cost of devices such as head mounted displays and data gloves 2

3 Basic Question Is it feasible to use a data glove to manipulate 3D objects in the user s normal desktop environment? 3

4 Overview Technology Methodology User studies Feature extraction ART2 NN Application Concluding remarks 4

5 What is a Data Glove? Data gloves measure hand gestures by sensing the bending of finger joints and finger separation. 5

6 5DT Data Glove 14 Ultra Developed by Fifth Dimension Technologies and is made of a lightweight stretch lycra material. 14 bend sensors use the fiber optic based flexor technologies Emits a stream of raw sensor values 6

7 What is a Flock of Birds? Flock of Birds (FOB) measures motion tracking of position and orientation of a sensor by a transmitter. 7

8 Flock of Birds Developed by Ascension Technologies and is a six degrees-of-freedom measuring device. Bird unit Contains its own independent computer Transmitter Transmits a pulsed DC magnetic field that is measured by the sensor Sensor Computes its position and orientation 8

9 Methodology Perform user studies to get training data Perform feature extraction on training data Train a neural network to recognize grasping objects Develop a simple application to test effectiveness of grasp and motion system 9

10 User Studies Take a variety of users through a choreographed set of motions which can then be used in supervised learning. Use a neural network as the learning mechanism because of the great variability among users. 10

11 User Study Architecture File System sampling information Database user information Data Acquisition data Server data Client data 3rd GUI commands C/C++ Application Java Application sensor values sensor values 80Hz Data Glove Flock of Birds 11

12 3rd GUI 12

13 User Trial Interaction User sits in front of computer wearing a data glove and FOB sensor while interacting with 3rd GUI Performs different types of hand and arm motions as directed Varying speeds Different positions and orientations Arbitrary motions Each trial duration is approximately 1 minute 13

14 Trial Types Trial Type 1: Collect grasping information Gestures with the glove Open/close hand Trial Type 2: Collect tracking information Trace object on screen Map physical location of users hand to a position on screen 14

15 Feature Extraction Select meaningful features that are lowdimensional and capture the underlying coordination of manipulative hand movements. Miriam Zacksenhouse & Paul Marcovici,

16 Feature Extraction (cont.) Manipulative hand movements Coordinated movements of the fingers to manipulate an object within the hand Classified as simultaneous or sequential movements Simultaneous hand movements Simple: participating digits flex and extend concurrently Reciprocal: different digits flex while others extend Coordination provides an inherent representation of the movement Underlying coordination: in-phase or anti-phase Projection of coordination in joint space results in straight-line trajectories Structure of the coordination is captured by the direction of the straight line 16

17 Example of time domain Index Joint Thumb Joint sensor values sensor values time time 17

18 Example of joint-space Index v.s. Index Index v.s. Thumb sensor values sensor values sensor values sensor values 18

19 Feature Extraction Approach Given the training data, compute a vector for each sensor containing the corresponding sensor values Identify most active sensor/joint Contains the vector of slopes Make corresponding vector of slopes available for classification Reduce amount of information needed for classification 19

20 Feature Extraction Architecture Database sensor values Feature Extraction MATLAB application most active joint answers ART2 NN recognized gesture 20

21 Joint Space Trajectory - Pinch 21

22 Adaptive Resonance Theory (ART) Used in pattern-classification problems Solves the stability-plasticity dilemma Instabilities arise from irrelevant input Adapt to novel input Feedback mechanism Resonant state Has previously learned to recognize an input pattern New pattern is stored for the first time Carpenter & Grossberg,

23 Benefits Learn new categories with single novel input Adapt to changes without forgetting previously learned patterns Supports on-line adaptation Eliminates the need for large training data sets and off-line training 23

24 ART1 versus ART2 ART1 uses binary input ART2 uses analog input 24

25 ART2 Architecture 25

26 Offline Training Architecture Database sensor values Feature Extraction MATLAB application most active joint answers ART2 NN recognized gesture 26

27 Application 3rd Application Take recognized movement and map the hand gesture to a command Grasp, ungrasp, zoom, etc. Take the position and orientation and map the hand motion and speed to the virtual object Translate, rotate, etc. 27

28 Application Architecture 3rd Application recognized gesture ART2 NN most active joint (X,Y,Z) position Feature Extraction Data acquisition sensor values raw sensor data Flock of Birds raw sensor data Data Glove 28

29 Conclusion Determine the feasibility of using a data glove for 3D object manipulation in a desktop environment Develop a user study to get training data Perform feature extraction Use the ART2 NN to recognize hand gestures Develop a simple application to test the efficiency of hand gestures and motions 29

30 References Zacksenhouse M, Marcovici P. Interactive recognition of simultaneous manipulative hand movements. Mechatronics ; Zacksenhouse M, Marcovici P. Inherent structure of manipulative hand movements and its discriminative power. Intelligent Robots and Systems ; Freeman JA, Skapura DM. Neural networks, algorithms, applications, and programming techniques. Reading: Addison-Wesley, Elliott J.M. and K.J. Connolly. A classification of manipulative hand movements. Developmental Medicine & Child Neurology ;

31 Questions 31