3D Data Navigation via Natural User Interfaces Francisco R. Ortega PhD Candidate and GAANN Fellow Co-Advisors: Dr. Rishe and Dr. Barreto Committee Members: Dr. Raju, Dr. Clarke and Dr. Zeng GAANN Fellowship Supervisor: Dr. Milani
The most profound technologies are those that disappear The computer for the 21st century Mark Weiser, 1999 Father of ubiquitous computing
WIMP Windows Icons Menu Pointers Paradigm Most successful input paradigm
Bill Buxton Three State Model Stylus Example
Bill Buxton Three State Model Mouse Example
Fitt s Law
Post-WIMP The set of components that will be required for the correct interaction in this new paradigm: Multi Touch Vision Tracking Speech Recognition Pen Based Interaction Virtual Devices Etc.
NUI and Other Devices
Software Visual Studio 2010 C++ Ogre3D OpenGL Windows 7 Additional libraries
Motivation Find methods that allow users to interact with technology seamlessly Add building blocks to Post-WIMP Contribute a easy adaptable framework to software without full gesture support Help domain specific problems via MT Neural Sciences (e.g., MRI)
Example: 3D-2D-3D
Domain Specific Example MRI 3D and MRI data slices
BACKGROUND Understanding Touch Multi-Touch Interaction Gesture Recognition Virtual Devices Design Guidelines Current Open Systems / Frameworks
Understanding Touch Interactions Oblique finger orientation w/o sensors Rotations and Translations Combined or separate? One or two hands One hand better for simpler tasks (rotations) Two hands better for separable type of tasks (anchor gestures)
Multi-Touch Techniques Hancock et el, 2007 One two three touches Algorithm
Gesture Recognition Methods Hidden Markov Models Sezgin and Davis, 2005. Neural Networks Pittman, 1991. Featured-Based Classifiers Rubine, 1991. Template Matching Kara and Stahovic, 2005. Geometric Recognizers Wobbrock and Wilson, 2007 Etc.
Gesture Recognizer Example The $1 and the $N algorithms are examples of a combination of : Geometric Recognizers Template Matching
The $1 Algorithm (Wobbrock and Wilson, 2007) Resample the points in the path (N=64) Find indicative angle between the centroid and gesture s first point Rotate gesture to Indicative angle = 0 Scale gesture to reference square Calculate the distance between: Candidate gesture to each stored template
The $N Algorithm (extension to $1) Anthony and Wobbrock, 2010
Virtual Devices New set of devices that either: Emulate Physical Devices Find new ways for interaction that are not possible with current physical devices Examples Virtual Keyboard Virtual Sphere
Virtual Keyboard
Virtual Sphere Chen et el. 1998 Simulates a 3D Trackball X axis: Left-Right Y Axis: Top-Down Z Axis: Circular Fashion *
Design Guidelines I (Bowman et el. 2005) Use existing manipulation techniques unless an application will benefit greatly from creating new ones. Swipe Gesture Perform task analysis when selecting a 3D interaction technique. How precise was the Rotation? Match the interaction to the device. Consider the specific device properties
Design Guidelines II (Bowman et el. 2005) Reduce wasted motion. Amplify the user hand motion Reducethe number of degrees of freedom (DOF) whenever possible. Constraint 3D interaction when possible There is no single best 3D manipulation technique
Set-up Methods / Implementation Main Display 3M 22 M2256PW Monitor Windows 7 MT Drivers and Library Recognition and Navigation Algorithms
Camera Setup
Gestures [GestureWorks] Anchor Rotate Two Hand (5) Divide 4 Finger Rotate 5 Finger Hold
Basic Gestures
Gesture Action: Gesture Rotation
Beyond Basic
Point of Clouds
How does it work? Receive Trace Collection from Operating System Generic Touch Events: Up, Move, Down events Detect Trace Event Uses a MAP to keep collections of traces Tap, Double Tap and Drag Delegate responsibility to Trace Manager Trace Manager is aware of different recognizers Allow event and poll techniques
How does it work? (cont.) Decision Manager Collects possible gestures from recognizers Using a priority criteria, selects Gesture Action Manager Selects action with correct values Name of Gestures Callback function (or raised event) Transformation Matrix
Overview of the Framework System Diagram A system level view of our proposed framework Class Diagram The current classes that are part of our framework MT FSM Multi-Touch Finite State Machine
System Diagram
Evaluation Pilot Program (3-5 subjects) Correct bias produced by Instruments Experimental procedures Environmental Factors **
Main Experiment Main Experiment (30 Subjects) Repeated Measure design Compares Our approach to an existing Approach Will measure: Execution time: measures gesture actions to accomplish task e.g. : User will navigate within a 3D maze Accuracy of the movement: measures steps to finish What was the path taken to accomplish task? Likert scale: measures usability of the user
Preliminary Results
Preliminary Results Support for other devices for comparison Wiimote 3D Mouse Gamepad
Questions For references, please email me to forte007@fiu.edu Thank you