Occlusion based Interaction Methods for Tangible Augmented Reality Environments Gun A. Lee α Mark Billinghurst β Gerard J. Kim α α Virtual Reality Laboratory, Pohang University of Science and Technology β Human Interface Technology Laboratory New Zealand, University of Canterbury endovert@postech.ac.kr POSTECH Univ. of Canterbury
Introduction Tangible Augmented Reality [Kato 2001] Each virtual object registered to physical object Users manipulate virtual objects by manipulating corresponding physical objects 2
Introduction Tangible AR Applications Tiles VOMAR MagicBook ARGroove 3
Introduction Tangible Augmented Reality Easy natural 3D interactions Full 6DOF 3D Direct manipulation 1/2D interactions? Buttons? Sliders? Menus? Keypads? 4
Related Works 2D Input methods in AR Screen stabilized fashion Tinmith [Piekarski 2002] Magic Ring [Dias 2003] World stabilized fashion Windows on World ARGUI [Geiger 2003] Hand gesture mouse [McDonald 2003] Used indirect input methods (mouse, getsture, etc) rather than direct touch 5
Related Works Large scale touch sensor SmartSkin [Rekimoto 2002] Touch implies occlusion Commercial products using occlusions as inputs Virtual Keyboard, Canesta Parapara Paradise, KONAMI 6
Occlusion based Interaction 2D Interaction using pointers Pointer centered view Tracking the pointers Cost depends on number of pointers + interaction points Interaction point centered view Sensing on interaction points Cost depends on number of interaction points In TAR environments, interaction points (props) are tracked rather than pointers (hands). 7
Occlusion based Interaction Occlusion Detection in TAR Tracking a set of visual markers with predefined spatial relationship Provides robustness Invisible marker positions are inferred from the visible ones Occlusion = Invisible + Within the view Two methods for detecting occlusions Boundary Marker Method Estimated Marker Projection Method 8
Occlusion based Interaction Boundary Marker Method 9
Occlusion based Interaction Estimated Marker Projection Method 10
Occlusion based Interaction Boundary Marker Method Easy and straight forward, reliable implementation Lesser computation Estimated Marker Projection Method No need of boundary markers More computation required Estimation errors due to lens distortion Over estimating the size of markers helps 11
Occlusion based Interaction Interaction Design Time out constraints Point & hold -> Explicit command Preventing accidental commands Repeating the command (common in keyboards) 12
Occlusion based Interaction Interaction Design (cont d) Sub-marker level measurement Two consecutive markers occluded -> The pointer is in-between them Relatively lesser markers needed 13
Occlusion based Interaction Interaction Design (cont d) Tip point marker detection 2D grid of markers Top-left one from the occluded marker blob Other heuristics applicable e.g. select top-right one when the blob is skewed to the right 14
Implementation Desktop Custom built PC with Windows XP Athlon 1.5GHz / 512MB NVIDIA GeForce4 MX Mobile Apple Macintosh ibook with Mac OS X G3 900MHz / 640MB ATI Radeon 7500 15
Implementation ARToolKit (2.65DS / 2.61Mac) Logitech Notebook PRO USB Camera 320x240 @ 30Hz i-visor HMD Video image stretched to fit 800x600 OpenGL drawn in full resolution Performance: 19~30fps 5fps with 35 markers on mobile configuration (low computation power) 16
Applications Buttons 17
Applications Sliders 18
Applications Menu bars 19
Applications Ball-pushing Game 20
Applications Calculator 21
Applications Board games 22
Discussion Informal user study 6 subjects 7 scale questionnaire Easy to learn: 5.6 (stddev=0.55) Easy to use: 5 (stddev=0.7) Two 2D game applications demonstrated on public Over 100 peoples tried and gave positive feedbacks 23
Discussion The interaction appeared natural Using bare hands Direct manipulation (pointing) Occlusion of users hands by virtual objects were reported unnatural Partially solved by making the interface semitransparent when occluded Easily applicable to mobile and/or wearable systems 24
Conclusion & Future Works Occlusion based Interaction for Tangible AR Environments Simple, easy to use, and natural 2D interaction Low development and computational cost Future Works Try other type of visual markers or visual tracking methods Add passive tactile feedbacks 25