Input devices and interaction. Ruth Aylett

Transcription

1 Input devices and interaction Ruth Aylett

2 Tracking What is available Devices Gloves, 6 DOF mouse, WiiMote, Kinect Contents

3 Why is it important? Interaction is basic to VEs We defined them as interactive in real-time No interaction => NOT a VE Ideal interaction: Very low latency - i.e fast Multi-modal Unencumbered Intuitive Technology falls well short of this of course

4 Tracking the human body Large displays require position and orientation of viewer s body to be tracked tracking information fed to runtime system as input signal. Most commonly tracked is head but sometimes also hands, arms, legs, eyes etc. Head tracking used to update virtual viewpoint orientations. Body tracking needed for lifelike interaction with objects and creatures. say user wishes to wave at another person in the VE: their real-world motions can be tracked and replicated in the VE.

5 Interaction types Navigation Staying on the ground? Walking v flying Depends on size of model wrt display system Degree of immersion Interaction with other users Gesture Interaction with objects Depends on the object and interaction Select, lift, rotate, throw, steer, hit

6 Virtual Tennis Movie Virtual Tennis

7 Tracking the human head An essential basic requirement in immersive VR systems. Imagine axes mounted on top of your head pans, tilts and yaws of head measured around those axes. HMDs rotation sensors measure these three angles. Angles passed to run-time VR software which updates viewing angles. to HMD

8 Tracking devices Many tracking devices and systems developed over the years some aimed specifically at VR systems others borrowed from other areas. Some systems are portable and cheap - some require permanent installations in large rooms and are very expensive indeed. Trackers can be magnetic, electro-magnetic, acoustic, inertial, optical, or mechanical.

9 Mechanical trackers Mechanical linkage system arm-like structure of several joint, one end fixed, the other free to move with the user. Measure position and angular orientation of free end by measuring angles at each joint and factoring in length of each segment. Fake Space BOOM (right)

10 Mechanical Tracking Advantages Simple sensors, no need for transmitter/receiver low-cost device very low latency High positional accuracy Disadvantages The user is tethered Lots of inertia Typically small working volume Mechanical parts wear out

11 Exoskeletons For bodies or parts of bodies Derived from assistive technology Working backwards

12 Electro-magnetic trackers transmitter generates electromagnetic signals received by a receiver (or sensor). Signal strength used to determine absolute position and orientation of receiver relative to transmitter.

13 Example: Polhemus FASTRAK FASTRAK electro-magnetic sensor from Polhemus accurately computes the position and orientation of tiny receiver as it moves through space. Dynamic, real time six degree-of-freedom measurement of position (X, Y, and Z) and orientation (yaw, pitch, and roll) RS-232 signal updated at 120 records/sec. Transmitter constantly puts out a weak magnetic field. passive receiver generates an electric signal as it is moved through the field. Polhemus' processing electronics then amplify and analyse this signal to determine the real-world position and orientation of the receiver relative to the transmitter.

14 Polhemus FASTRAK system Polhemous trackers well proven and widely used since the very early 1990 s. The FASTRAK system shown here has one receiver and one transmitter. System expanded by adding up to three more receivers can attach receivers to different parts of body log data for gait and limb analysis or computer animation.

15 Electromagnetic Tracking Polhemus

16 Electromagnetic Tracking Ascension Ascension market a number of systems based on DC rather than AC fields including Flock of Birds and a full gait analysis system called MotionStar.

17 Electromagnetic Tracking Advantages Small receivers Reasonably cheap Line-of-sight (LOS) not required Disadvantages Accuracy diminishes with distance Not very large working volume High latency due to filtering Transmitter/receiver required

18 Electro-magnetic interference Major problem of electro-magnetic trackers magnetic fields easily affected by the surrounding environment. Large metal objects produce eddy currents in the presence of the magnetic fields These can interfere and distort the original signal causing inaccurate measurements. same effect appears near electric currents, such as in cabling also ferromagnetic materials Also electromagnetic sources such as computer monitors. Ferromagnetic and/or metal surfaces cause field distortion

19 Ultrasonic trackers Two main components transmitter generating an ultrasound signal receiver detecting the signal. Distance is calculated by measuring time-of-flight of ultrasonic pulse. Three transmitters and receivers needed to calculate full 3D position and orientation. Ultrasonic tracking used by Logitech Head Tracker (shown) and 3D mouse.

20 Ultrasonic trackers The Power Glove made by toy company Mattel (who make Barbie) introduced in 1989 for use with the Nintendo Entertainment System (NES). Ultrasonic device for use in place of standard Nintendo controllers Detected finger motion Plus full set of buttons on the wrist. In fact not much use for Nintendo gamers But amazingly advanced piece of VR kit for its time.

21 Acoustic Tracking Advantages Well known transducers (mics), lightweight Low cost device Disadvantages Line-of-sigh (LOS) required Echoes Low accuracy (speed of sound in air varies) Transmitter/receiver required

22 Inertial tracking systems Very popular (because cheap) based on inertial gyro technology Detects acceleration and thus can calculate velocity (since mass in known) giving 3DoF Newish example is the Intersense IS-300. Can be coupled with add-on ultrasonic system to give 6 DoF sensing example of a hybrid technology tracker. IS-300 can operate in metallic environments, 6 DoF tracker operates only in LoS of transmitter. Other examples: Intersense Intertrax2 and the Ascension 3D-Bird.

23 Advantages Cheap Small size Inertial Tracking No transmitter/receiver required LOS not required Disadvantages Only 3 DOF on their own Drift Not accurate for slow movements

24 Optical tracking methods Many different forms Often use image processing and pattern recognition and matching Much work outside of VR: numerous ideas suitable for tracking object position and pose For example fiducial mark detection light sources or reflective colour markers attached to object at important locations such as joints or extremities. Easier for image processing algorithm to track in cluttered conditions.

25 How it is done

26 Optical tracking methods Outside-in tracker tracking apparatus is fixed object to be tracked (e.g. the user) is viewed from the "outside". Inside-out systems take tracking measurements from the object to be tracked for instance a camera can be mounted on the HMD images analysed to produce pose and distance estimations based on the position of fixed patterns within the environment. Visible images or infra-red used. Many optical systems (but not all!) are one-offs, expensive and require careful calibration procedures.

27 Motion Capture (Mo-cap) Originally developed for gait analysis Artificial limbs Taken up by film/animation industry Artificial actors (Titanic, LoR, etc..) Can also be used in real-time

28 Infra-red cameras

29 Placement of markers Depends on application Body/face? Occlusion issues? Attaching markers Body suit CMU database example Sweat bands etc Also in processing software

30 Occlusion problems Need a skeleton to find a lost marker that reappears Included with camera sw packages But not always in datastreaming mode

31 Optical Tracking Advantages Can work over a large area. Inherently wireless Disadvantages LOS needed Transmitter/receiver required Expensive Requires computer vision technology

32 Speckled computing Specks: small computing devices Eventual aim: like dust motes Present size: a few cm Supports inside-out tracking University of Edinburgh

33 Unencumbered tracking Depends on identifying hand/hand on video One approach is using blobs

34 Kinect Developed for xbox Sold 8 million units in 60 days from Nov 2010 launch Almost at once used for lots of other things E.g Spec: Colour and depth-sensing lenses Voice microphone array Tilt motor for sensor adjustment Field of View Horiz: 57 degrees Vert: 43 degrees Tilt range: ± 27 degrees Depth sensor range: 1.2m - 3.5m

35 What is in it Projects an IR grid Affected by other IR sources (e.g. sunlight) Built-in facial and voice recognition simultaneously tracking of up to six people, two active players with RT feature extraction of 20 joints per player

36 Interfacing with Kinect Originally unofficial use Driver Interface: Two buffers rgb-data depth-map (10bit resolution) From June 2011 official Windows 7 SDK

37 Interaction devices Ruth Aylett

38 Eye trackers Eye tracking systems are examples of optical tracking devices. viewpoint in the virtual world follows the gaze of user s eye. Originally developed as a mouse replacement simply look at object interact through eye movement (such as a slow blink). Support physically impaired users. Combined eye and head tracking systems also exist - use in practice is complicated.

39 Cybergloves and similar Inherent in the folklore and hype of VR is the cyberglove - a wearable device that monitors the the position and orientation of hand and fingers. The name CYBERGLOVE is registered by Virtual Technologies Inc (VTi). uses 18 or 22 patented angular sensors for tracking the position of fingers and hand.

40 Virtual Technologies CyberGlove - 18-sensor model - 22-sensor model Variants are: - CyberTouch - CyberGrasp Gloves

41 Gloves Fifth Dimensions Technologies - Data Glove Data Glove finger flexure hand orientation -roll & pitch

42 Gloves Fakespace - Pinch Glove Pinch Glove gesture recognition reliable low cost electrical sensors in each fingertip contact among any 2 or more digits

43 Mouse as input device in VR Normal 2D mouse can be used (as in Octaga for example). Need user selectable modes to switch between DoF s. More sophisticated mice provide 3 or more DoF: these include the Spaceball (shown here) and Spacemouse. Standard games joysticks or gamepads also used to give 2 or more DoF s.

44 6 DOF Mice 3 translation DOF 3 rotation DOF

45 6 DOF Mice Spaceball by Labtec Spacemouse by DLR (Logitech - USA)

46 6 DOF Mice Cyberpuck SpaceOrb

47 3 accelerometers Enough for 6 DOF But will drift Bluetooth connection to 10m Optical (IR) sensor To 5m from sensor bar Triangulation from ends of bar Allows accurate pointing Speaker The WiiMote

48 WiiMote interaction Head-tracking WiiMote stationary, head-mounted IR source Finger-tracking - touch-free interaction IR tape on finger + fixed IR source Gesture recognition Using accelerometers Feature classification Fast movements work better; beware variable arm orientation

49 Software Free libraries WiiGLE WiiGLE/doku.php Provides a set of classifiers WiiGee Java-based, one classifier Issues with Bluetooth stacks Flakey implementations, especially Vista BlueSoleil seems a good driver

50 Interaction issues Types: Navigation Selection Manipulation System control

51 Navigation Move between locations: simplest form of interaction Explore the environment; look at it Ambiguous term: Path integration/dead reckoning Wayfinding Locomotion

52 Path integration/dead reckoning From known coordinates: How far have I come? Distance/rotation So where am I now? Continuous update cycle Compare with desired end coordinates

53 Wayfinding What route from here to there? Requires spatial knowledge/understanding Of landmarks, routes A natural approach? Line-of-sight; Virtual signage; virtual map If used for training for example Sound? Floating arrow? Head-up display? Naturalism works better for direction than distance

54 Process Orientation Where am I wrt to nearby objects and the target location? Route Decision Choose route that does go to destination. Route Monitoring Check on the correct route and going in the right direction. Destination Recognition Are we there? Or at least close?

55 Locomotion Teleportation Scene In Hand Eye In Hand Flying Vehicle Manipulation methods as locomotion Leaning (relative vs absolute) Speech driven Device Specific (driving)

56 Movement Physical locomotion User physically moves within tracker coverage area Frame of reference for interaction is own body Real-world orientation cues Virtual locomotion Virtual world moves/rotates around user Frame of reference is display device/physical frame Can cause disorientation quicker then physical motion

57 Locomotion styles Unconstrained Tracker orientation determines direction of travel Users may roll or go up/down as they move forward-left-rightback May disorient new users; difficult to control Users travel through objects;no collision detection High-speed navigation,needs high frame rate Constrained Restricts locomotion to plane/direction (no roll is common) Sets some speed/acceleration limits (may take frame rate into account) Collision detection adds computational load to application Better orientation cues Enhances solidity & realism of environment

58 Selection Direct manipulation Intuitive; can be tiring; or infeasible Ray-casting Laser pointer; less tiring May be imprecise: small objects, distances, tracker noise Manipulation? Cone-casting (spotlight) Easier than ray-casting May select multiple objects Also manipulation issue