3D Data Navigation via Natural User Interfaces

Size: px

Start display at page:

Download "3D Data Navigation via Natural User Interfaces"

Roy Thornton
5 years ago
Views:

1 3D Data Navigation via Natural User Interfaces Francisco R. Ortega PhD Candidate and GAANN Fellow Co-Advisors: Dr. Rishe and Dr. Barreto Committee Members: Dr. Raju, Dr. Clarke and Dr. Zeng GAANN Fellowship Supervisor: Dr. Milani

2 The most profound technologies are those that disappear The computer for the 21st century Mark Weiser, 1999 Father of ubiquitous computing

3 WIMP Windows Icons Menu Pointers Paradigm Most successful input paradigm

4 Bill Buxton Three State Model Stylus Example

5 Bill Buxton Three State Model Mouse Example

6 Fitt s Law

7 Post-WIMP The set of components that will be required for the correct interaction in this new paradigm: Multi Touch Vision Tracking Speech Recognition Pen Based Interaction Virtual Devices Etc.

8 NUI and Other Devices

9 Software Visual Studio 2010 C++ Ogre3D OpenGL Windows 7 Additional libraries

10 Motivation Find methods that allow users to interact with technology seamlessly Add building blocks to Post-WIMP Contribute a easy adaptable framework to software without full gesture support Help domain specific problems via MT Neural Sciences (e.g., MRI)

11 Example: 3D-2D-3D

12 Domain Specific Example MRI 3D and MRI data slices

13 BACKGROUND Understanding Touch Multi-Touch Interaction Gesture Recognition Virtual Devices Design Guidelines Current Open Systems / Frameworks

14 Understanding Touch Interactions Oblique finger orientation w/o sensors Rotations and Translations Combined or separate? One or two hands One hand better for simpler tasks (rotations) Two hands better for separable type of tasks (anchor gestures)

15 Multi-Touch Techniques Hancock et el, 2007 One two three touches Algorithm

16 Gesture Recognition Methods Hidden Markov Models Sezgin and Davis, Neural Networks Pittman, Featured-Based Classifiers Rubine, Template Matching Kara and Stahovic, Geometric Recognizers Wobbrock and Wilson, 2007 Etc.

17 Gesture Recognizer Example The $1 and the $N algorithms are examples of a combination of : Geometric Recognizers Template Matching

18 The $1 Algorithm (Wobbrock and Wilson, 2007) Resample the points in the path (N=64) Find indicative angle between the centroid and gesture s first point Rotate gesture to Indicative angle = 0 Scale gesture to reference square Calculate the distance between: Candidate gesture to each stored template

19 The $N Algorithm (extension to $1) Anthony and Wobbrock, 2010

20 Virtual Devices New set of devices that either: Emulate Physical Devices Find new ways for interaction that are not possible with current physical devices Examples Virtual Keyboard Virtual Sphere

21 Virtual Keyboard

22 Virtual Sphere Chen et el Simulates a 3D Trackball X axis: Left-Right Y Axis: Top-Down Z Axis: Circular Fashion *

23 Design Guidelines I (Bowman et el. 2005) Use existing manipulation techniques unless an application will benefit greatly from creating new ones. Swipe Gesture Perform task analysis when selecting a 3D interaction technique. How precise was the Rotation? Match the interaction to the device. Consider the specific device properties

24 Design Guidelines II (Bowman et el. 2005) Reduce wasted motion. Amplify the user hand motion Reducethe number of degrees of freedom (DOF) whenever possible. Constraint 3D interaction when possible There is no single best 3D manipulation technique

25 Set-up Methods / Implementation Main Display 3M 22 M2256PW Monitor Windows 7 MT Drivers and Library Recognition and Navigation Algorithms

26 Camera Setup

27 Gestures [GestureWorks] Anchor Rotate Two Hand (5) Divide 4 Finger Rotate 5 Finger Hold

28 Basic Gestures

29 Gesture Action: Gesture Rotation

30 Beyond Basic

32 Point of Clouds

33 How does it work? Receive Trace Collection from Operating System Generic Touch Events: Up, Move, Down events Detect Trace Event Uses a MAP to keep collections of traces Tap, Double Tap and Drag Delegate responsibility to Trace Manager Trace Manager is aware of different recognizers Allow event and poll techniques

34 How does it work? (cont.) Decision Manager Collects possible gestures from recognizers Using a priority criteria, selects Gesture Action Manager Selects action with correct values Name of Gestures Callback function (or raised event) Transformation Matrix

35 Overview of the Framework System Diagram A system level view of our proposed framework Class Diagram The current classes that are part of our framework MT FSM Multi-Touch Finite State Machine

36 System Diagram

37 Evaluation Pilot Program (3-5 subjects) Correct bias produced by Instruments Experimental procedures Environmental Factors **

38 Main Experiment Main Experiment (30 Subjects) Repeated Measure design Compares Our approach to an existing Approach Will measure: Execution time: measures gesture actions to accomplish task e.g. : User will navigate within a 3D maze Accuracy of the movement: measures steps to finish What was the path taken to accomplish task? Likert scale: measures usability of the user

39 Preliminary Results

40 Preliminary Results Support for other devices for comparison Wiimote 3D Mouse Gamepad

44 Questions For references, please me to Thank you

Direct Manipulation. and Instrumental Interaction. CS Direct Manipulation

Direct Manipulation. and Instrumental Interaction. CS Direct Manipulation Direct Manipulation and Instrumental Interaction 1 Review: Interaction vs. Interface What s the difference between user interaction and user interface? Interface refers to what the system presents to the