Research Seminar Stefano CARRINO stefano.carrino@hefr.ch http://aramis.project.eia- fr.ch 26.03.2010
- based interaction Characterization Recognition Typical approach Design challenges, advantages, drawbacks Applications Conclusion 2
- based interaction, why? The gestures are a natural way to interact with object, tools and other people As substitution for other forms of communication when other interactions are not possible Impaired people Special context As complement to other types of interaction modalities 3
A motion of the limbs or body made to express or help express thought or to emphasize speech. The act of moving the limbs or body as an expression of thought or emphasis. An act or a remark made as a formality or as a sign of intention or attitude. A succession of postures. Own definition (for this seminar): An intentional sign made with the body or limbs to communicate intention or information 4
s Vs Gesticulation Also the gesticulation provides information Static Vs Dynamic s Static gestures (aka postures, poses, ) Dynamic gestures: a sequence of postures/positions Multi- dimensional gestures 2D gestures 3D gestures Pointing gesture 5
2D gesture 3D gesture Pointing gesture 6
- based interaction Characterization Recognition Typical approach Design challenges, advantages, drawbacks Applications Conclusion 7
Dynamic gesture recognition (through computer vision) can be divided in the following main phases: Detection Tracking segmentation recognition Features extraction Detection Tracking Segmentation Recognition Features Extractions 8
Two sub- steps: Image acquisition Preprocessing Detection Tracking Segmentation Features Extractions Recognition 9
Image acquisition Mono- camera, multi- camera, stereo- camera, or 3D camera Camera resolution (low Vs high resolution) Frames per second Detection Tracking Segmentation Features Extractions Recognition 10
Preprocessing Pixel level segmentation Color segmentation Hand detection Color marker detection Motion segmentation Background subtraction Works good on known background (static background) Cannot detect stationary hands or determine which moving object is the hand Detection Tracking Segmentation Recognition Features Extractions 11
Preprocessing Contour detection Not directly depending on skin color and lighting conditions Can be a large number of objects (even in the background) Correlation Problems when objects are rotated or scaled Problem can be avoided with continuously updating the template Detection Tracking Segmentation Recognition Features Extractions 12
Frame 1 1 Detection 2 Frame 2 1 Tracking Segmentation Recognition Features Extractions 2 13
Approaches Kalman filter Easily computable in real- time Basic form of Kalman filters cannot track objects on unknown background Condensation One of the most used technique for tracking Detect and track contour of moving objects in a cluttered environment CAMshift Fast, real- time It may be possible to improve accuracy by using different color representation There are quite a few parameters System without tracking In controlled environment with a special gesture vocabulary Detection Tracking Segmentation Recognition Features Extractions 14
segmentation Initial (final) posture When hands are not moving - > end of gesture decomposition Preparation, stroke and retraction Statistical approach Hidden Markov Model Detection Tracking Segmentation Recognition Features Extractions 15
Potential features: Position, acceleration, velocity Spatial temporal width FFT of the position The features can be extracted in three steps of the process chain Post- processing should be done before providing the features to the GR block Detection Tracking Segmentation Recognition Features Extractions 16
Classification Algorithms: Hidden Markov Model (HMM) Ergodic HMM, Left- Right HMM, Left- Right Banded Hierarchic HMM, Input- Output HMM, Parametric HMM, etc. etc. etc. Conditional Random Fields (CRF) Hidden CRF, Latent- dynamic Discriminative CRF, etc. Neural Networks (NN), Decision Trees, Support Vector Machine (SVN), KNN, Dynamic time warping (DTW), etc. Boosting Algorithms, etc. Hybrid algorithms Detection Tracking Segmentation Recognition Features Extractions 17
- based interaction Characterization Recognition Typical approach Design challenges, advantages, drawbacks Applications Conclusion 18
Design challenges Lighting conditions Provide a feedback to the user vocabulary (small large, kind of gesture used, etc.) Real- time interaction Wearable gesture interfaces Multimodality Skinput project 19
Advantages: Natural way of interaction Space effective interaction modality (compared with keyboard and mouse) Removes the user s dependency on a surface Remote interaction Drawbacks: Tiring (e.g. gorilla arm) User dependent gestures few universal understandable gestures Computationally expensive 20
Natal project The Project Natal sensor device Sixthsense G-speak 21
Project natal: http://www.xbox.com/en- US/live/projectnatal/ Oblong g- speak: http://www.oblong.com/ Sixth Sense: http://www.pranavmistry.com/projects/sixthsense/ Touchless: http://www.codeplex.com/touchless Wiiremotes (and soon the Play Station Move) 22
- based interaction Characterization Recognition Typical approach Design challenges, advantages, drawbacks Applications Conclusion 23
based interaction as interface characterization Gesticulation & gesture Dynamic Vs static gesture Multi- dimensional gesture Typical approach Challenges, advantages, and drawback 24
25