Outline 3D Interfaces Output Paulo Dias Interactive Computing Systems MAPi Doctoral Programme (06/01/2014) z 3D User interfaces Output Vision / Graphic displays Introduction Stereopsis Personal and Large Volume Display Audition Touch and Force Feedback Taste and Smell Overview of VR frameworks Interaction in 3D worlds: Navigation, Selection and Manipulation Demo: Oculus Rift Practical labs: introduction to Kinect SDK deti departamento de electrónica telecomunicações e informática universidade de aveiro 2 Output Devices Human Senses The human senses need specialized interfaces: Vision - visual feedback with Graphics displays Audition - 3-D audio localized sound Touch - Haptic interfaces Smell and Taste feedback - few examples based on palette of odorants and chemicals 3 4 The ultimate display? Don t think of that thing as a screen, think of it as a window, a window through which one looks into a virtual world. The challenge to computer graphics is to make that virtual world looks real, sounds real, moves and responds to interaction on real time, and even feels real. Vision / Graphic Displays A graphics display is a computer interface that presents synthetic world images to one or several users interacting with the virtual world. "The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. Personal displays: Main technologies: - HMDs - Binoculars LCDs/OLEDs - Monitor-based displays/active glasses LCDs - Autostereoscopic displays lenticular/barrier Large volume displays: Workbenches Caves projectors Walls Surface tables (Microsoft, ) (Ivan Sutherland, The Ultimate Display 1965) 5 6 1
Human Visual System Vision is the dominant sensorial channel Depth perception in mono images is based: on occlusion (one object blocks another from view) Shadows textures motion parallax (closer images appear to move more than distant ones) Human Visual System Many of the perceptual cues we use to visualize 3D structure are available in 2D projections Such cues include: occlusion (one object partially covering another) perspective (point of view) familiar size (we know the real-world sizes of many objects) atmospheric haze (objects further away look more washed out) Four cues are missing from 2D media: stereo parallax seeing a different image with each eye movement parallax seeing different images when we move the head accommodation the eyes lenses focus on the object of interest convergence both eyes converge on the object of interest 7 8 Stereopsis Stereo ="solid" or "three-dimensional opsis = appearance or sight Also: binocular vision, binocular depth perception stereoscopic depth perception object Projection plane Stereopsis is the impression of depth that is perceived when a scene is viewed with both eyes by someone with normal binocular vision Binocular disparity is due to the different position of our two eyes eyes Stereopsis Depth perception in stereo is based on stereopsis: when the brain registers and fuses two images Image parallax means that the two eyes register different images (horizontal shift) The amount of shift depends on the interpupillary distance (IPD) (varies for each person in the range of 53-73 mm) Works in the near field (to a few meters from the eye) 9 10 Stereopsis Stereopsis: implications for Graphic devices Need to present two images of the same scene (Hearn and Baker, 2002) object The two images can be presented: at the same time on two displays Left eye, right eye images (HMD) time-sequenced on one display (active glasses) Right eye image Left eye image eyes Projection plane spatially-sequenced on one display (auto-stereoscopic displays) 11 12 2
Common ways to produce 3D sensation Anaglyphs: two colored images and color coded glasses (red/cyan(green)) Common ways to produce 3D sensation All show the right image to the right eye and the left image to the left eye! Two images with different light polarization and polarizing glasses Linear and circular Double frame-rate displays combined with LCD shutter glasses Autostereoscopic displays Parallax barrier and lenticular lens All these technologies provide: stereo parallax When combined with headtracking, they can provide movement parallax for a single viewer Head Mounted Displays (HMDs) and exotic displays 13 14 Graphic Displays LCDs (Liquid Crystal Displays) Layers in a LCD States of a LCD Graphic Displays Organic LEDs (OLED) In an active-matrix OLED display, each pixel element can stay on during the entire frame time There are no intrinsic limitations to the pixel count, resolution, or size of an active-matrix OLED display A defective pixel produces a dark effect, which is less objectionable than a bright point defect (as in LCD s) Colour LCD 15 16 Graphic Displays Organic LEDs (OLED) OLEDs advantages: Robust Design Viewing Angles High Resolution Production Advantages Video Capabilities Hardware Content Power Usage Graphic Displays Personal Display A graphics display that outputs a virtual scene destined to be viewed by a single user. Such image may be monoscopic or stereoscopic, monocular (for a single eye) or binocular (displayed on both eyes). Head Mounted Displays (HMDs) 3-D Binoculars (hand supported) Auto-stereoscopic displays (desk supported) 17 19 3
Graphic Displays HMD integration in a VR system Graphic Displays HMD examples Simple HMD zsight OLED professional HMD (Sensics) - Bright, crisp, high-contrast OLED display - Stereo - High resolution full-color SXGA 1280 1024 pixels Professional HMD - 60º field of view - Integrated yaw/pitch/roll tracker - Self contained, lightweight (450 g) - Integrated high-quality stereo audio and microphone - Price ~$13000 http://sensics.com/products/head-mounted-displays 21 22 Graphic Displays HMD examples Vuzix 920AR MaxReality Twin high-resolution 640 x 480 LCD displays Equivalent to a 67-inch screen (at ~ 3 m) 60 Hz progressive scan update rate Ultra-low video distortion 31-degree diagonal field of view 24-bit true color (16 million colors) Independent +2 to -5 diopter focus adjustment Two discrete VGA (640 x 480) video cameras 30 frames/s video capture 640 x 480 USB video camera no proprietary drivers required 6 DOFs tracker yaw, pitch, roll, and x, y, z 3-axes magneto-resistive sensors 3-axes accelerometers 3-axis gyros Auto re-centering to adjust from drift Includes tracker and gyro calibration software Price: ~$1500 http://www.vuzix.com/consumer/product Graphic Displays HMD examples HMZ-T1 Personal 3D Viewer (Sony 2011) Display Device - OLED Panel 2 Display Resolution 1280 x 720 Horizontal viewing angle- 45º Aspect Ratio - 16-9 Gradation - RGB 24bit Price ~800$ http://inition.co.uk/3dtechnologies/sony- 3d-hmd-hmz-t1 23 24 Graphic Displays HMD examples Graphic Displays Virtual Binoculars Oculus Rift Oculus Rift Price: $300 Display Resolution: 640 800 (1.25:1) [1280x800 split between each eye] Display Technology: LED Display Size (diagonal): 4.08 Field of View (degrees): H: 90 Pixels Per Inch (PPI): 251.1 Total Pixels: 512,000 Weight (headset): 220g Stereoscopic 3D capable: Yes Audio: Bring Your Own Audio Inputs: DVI, USB (power) Head Tracking: Yes Handheld, interactive, immersive displays combine high-resolution microdisplays with adjustable, wide field-of-view optics Have a familiar form (pair of binoculars) May have a tracker as accessory Examples: Virtual Binocular SX Virtual Binocular SV http://www.oculusvr.com/ 25 26 4
Graphic Displays Autostereoscopic displays Two technologies: Lenticular barrier Do not require use of special glasses Allow several vantage points Passive - do not track user s head (restrict user s position) Active - track the head motion (give more freedom) Virtual Binocular SX Price: $20,900 Virtual Binocular SV Price: $7,900 27 28 Graphic Displays Autostereoscopic displays Lenticular: an array of cylindrical lenslets is placed in front of the pixel raster Graphic Displays Autostereoscopic displays Parallax barrier: a barrier mask is placed in front of the pixel raster lenslets direct the light from adjacent pixel columns to different viewing slots at the ideal viewing distance Each of the viewer s eyes sees light from only every second pixel column Each of the viewer s eyes sees light from only every second pixel column 29 30 Graphic Displays Autostereoscopic displays Virtual Window - 19 Graphic Displays Autostereoscopic displays SynthaGram 404 StereoGraphics Co Key Features: - Large 19 diagonal screen - 3D mode without glasses - Digital (DVI-D) and analog (VGA) inputs - 1280 x 1024 resolution - Super fast 8ms response time - Price ~$4000 Optional Video Expansion Box Key features: - 40 LCD display - Resolution- 1280x768 pixels - 70 o horizontal viewing (7 to 15 feet) - Weight - 33.2 kg - Price - $12000 http://www.5dt.com/products/istg_synthagram_03.html http://inition.co.uk/3d-technologies/dimensiontechnologies-inc-virtual-window-19 31 32 5
Graphic Displays Autostereoscopic displays Graphic Displays Autostereoscopic displays Sharp autostereoscopic laptop Phillips BDL4251VS/00-107cm (42"), - Autostereoscopic 3D, - Full HD - Resolution: 1920x1080p - Price: 4000-15 diagonal display - 1024x768 resolution - 2D and 3 D mode - uses parallax barrier http://www.inition.co.uk/inition/product_stereovis_sharp_actius_rd3d.php 33 34 Graphic Displays Holographics displays Holographic displays The image is seen upon a nine optical layer glass panel Allow several co-located users to view a monoscopic or stereoscopic view of the virtual world - monitor-based large volume displays Objects appear to float in space http://www.youtube.com/watch?v=zr0y_sqlw04 - projector-based large volume displays Allow more freedom of motion vs. personal displays. http://www.eonreality.com/eon-holographic-i 38 Monitor-based Tiled monitors-based display Use active or passive glasses Several users can look at a monitor Can have a single monitor, or multiple side-by-side monitors If side-by-side, image continuity becomes an issue 39 45 6
Non-synchronized tiled image discontinuity Projector-based Large-Volume Displays Old technology is CRT-based (analog) three projector tubes (R, G, B) Synchronized tiled image Requires special fast green coating to avoid the fogging due to fast switching Suffer from low luminosity problems (200-300 lumens) (at 120 Hz) 46 47 Projector-based Large-Volume Displays Tilted surface Viewing Cone Technology makes transition from CRT-based (analog) to Digital Micromirror Device (DMD) (Digital Micro Mirror Device Display) projectors Reflector mirror - Cave-type display (CAVE, RAVE) - Wall-type displays - Domes Floor CRT projector (not shown) The old Fakespace ImmersaDesk workbench 48 49 CAVE 3-D large volume display (Fakespace Co.) CRT Projector Screen CAVE 3-D large volume display (Fakespace Co.) Mirror http://inition.co.uk/3d-technologies/fakespace-cave 50 51 7
Microsoft SURFACE one large display (projector) five infrared cameras tracks user s finger contact with the surface RAVE ( Re-configurable Virtual Environment ) Modular construction that allows various viewing configuration, from flat wall, to angled theater, to CAVE Vertical wall image 2.3 m X 2.4 m Several CRT projectors (260 lumens, 1280x1024 resolution) Takes 30 minutes or less to reconfigure 52 53 Wall-type displays Accommodate more users Using a single projector on a large wall means small image resolution Tiled displays place smaller images side-by-side they need multiple projectors Images need to have overlap, to assure continuity Overlap from two projectors means intensity discontinuity (brighter images in the overlap areas) Projectors need to modulate intensities to dim their light for overlap pixels 54 55 56 57 8
Tiled composite image from four projectors Tiled composite image from four projectors after adjustment 58 59 Video walls - LCD - LED rear projection -... Dome-type display Immersadome - 3D immersive visual environment - Up to 20 people (depending on dome size) - Resolution: 1400 x 1050-180 horizontal x 135 vertical (distortion free projection) - Multiple control options (mouse, keyboard, joystick, tracker-ball, haptic) Barco s sale 12000 - Applications: - museums, - group training - flight or driver simulation 60 http://inition.co.uk/3d-technologies/immersivedisplay-immersadome-range 61 Wall and Dome-type displays Advantages: - Accommodate more users (tens to hundreds) - Give users more freedom of motion Audition Importance of sound: Complementary information Alternative feedback Only real 3D sense Cannot be disconnect Disadvantages: - Large cost (up to millions of dollars) - Even with multiple projectors, resolution is low (as the area is large). 62 63 9
Stereo vs 3D Sound Audition 3-D audio should not be confused with stereo sound 3D Audio Displays Sound displays are computer interfaces that provide synthetic sound feedback to the user interacting with the virtual world. The sound can be - monoaural (both ears hear the same sound) or - binaural (each ear hears a different sound). 65 Sound synthesis Audio Applications for effective interaction has to met following criteria: 3D Localization Acoustics Simulation Speed and Eficiency Auralization Auralization: recreation of the acoustic environment. Reproduce the audio stimuli of the eardrum of the user for a given position in a given room. Direct Sound First reflections Late reverberation t 66 67 Auralization: AcousticAVE Project Auralization in closed environments Use of HRTF First Reflections with ray-tracing Reverberation Time RT 60 with several materials Visualization and auralization in Real Time in VR HRTFs Head Related Transfer functions Head-Related Transfer Functions (HRTFsdescribe the effect of head, torso and outer ear on the audio signals. Effect of the body from a given direction for a given head rotation can measured as the transfer function of a filter. HRTFs are measured with tiny Probes in dummy ears. VTK, sound and signal processing libraries (PortAudio, libsoundfile,...) 68 69 10
Early reflections Reconstrução 3D e Auralização: Early reflections are computed using virtual image,method. Each reflection considered as an additional source Wall Virtual Image method Computation of 6, first order reflections Semana Aberta da Ciência e Tecnologia da UA 70 71 Touch Touch Feedback Relies on sensors in and close to the skin Conveys information on contact surface geometry, roughness, slippage, temperature Does not actively resist user contact motion Easier to implement than force feedback Touch Force Feedback: Relies on sensors on muscle tendons and bones/joints proprioception Conveys information on contact surface compliance, object weight, inertia Actively resist user contact motion More difficult to implement than touch feedback 72 73 Human touch sensing mechanism Most touch sensors are on the hand (much less density on other parts of the body) Four primary types of sensors: 40 % are Meissner s corpuscles velocity detectors (detect movement across the skin) 25% are Merkel s disks measure pressure and vibrations 13 % are Pacinian corpuscles acceleration sensors deeper in skin (dermis) Most sensitive to vibrations of about 250 Hz 19% are Rufini corpuscles detect skin shear and temperature changes Human touch sensing mechanism Skin touch sensors 75 11
Sensorial adaptation Measure the decrease in electrical signals from the skin sensor over time, for a constant stimulus Slow Adapting (SA) - sensor produces a constant electrical discharge for a constant mechanical stimulus Rapidly Adapting (RA) - the rate of electrical discharge drops rapidly over time for a constant stimulus Spatial resolution Measure the receptive field size of a sensor Large receptive field sensor has low spatial resolution (Pacinian and Ruffini) SA-II, RA-II Small receptive field sensor has high spatial resolution (Meissner and Merkel) SA-I, RA-I 76 77 Spatial resolution Two-point limen test: 2.5 mm fingertip, 11 mm for palm, 67 mm for thigh Human grasping configurations 78 80 Touch Feedback Interfaces The ifeel Mouse (~2000, not in production) Can be desktop or wearable (gloves) Touch feedback mouse Force feedback joysticks CyberTouch glove Temperature feedback actuators Vibro-tactile actuator 81 12
CyberTouch Glove (Virtex) 6 individually Controlled Vibrotactile actuators 0-125 Hz frequency 1.2 N amplitude at 125 Hz Temperature feedback Added simulation realism by simulating surface thermal feel Single pump can produce 65 C differentials Uses thermoelectric pumps made of solidstate materials sandwiched between heat source and heat sink No moving parts 83 86 Force Feedback Interfaces Force Feedback Interfaces: Logitech 3D Need mechanical grounding to resist user motion Can be grounded on desk, wall, or on user body More difficult to construct and more expensive Than tactile feedback interfaces Uses potentiometers to sense position in spherical coordinates Uses electrical actuators to apply resistive torques ~$100 87 88 Force Feedback Interfaces: Geomagic Touch (former PHANToM Omni) Main application: medical simulations and training exercises in which the stylus emulates physical sensations (puncturing, cutting, probing or drilling) of using a syringe, scalpel, arthroscope or other instrument Other commercial, and scientific applications: - Robotic Control - Virtual Reality - Teleoperation - Training and Skills Assessment - 3D Modeling - Applications for the Visually Impaired - Entertainment - Molecular Modeling - Rehabilitation - Nano Manipulation,... Haptic devices vary according to workspace size, force, DOFs, inertia and fidelity http://www.youtube.com/ watch?v=0_nb38m86aw 89 Force Feedback Interfaces: CyberGrasp force feedback glove Force-reflecting exoskeleton that fits over a CyberGlove data glove Adds resistive force feedback to each finger Allows users to feel the size and shape of computer- generated 3D objects in a virtual world http://www.cyberglovesystems.com/products/ cybergrasp/photos-video 91 13
Example: SAMIRA : Interactive Maintenance Simulation with Haptic Feedback 92 http://www.3ds.com/fileadmin/products/3dvia/3dviavirtools/demo showcase/html/demo.html?br=1&rub=2&srub=8&de=151# 93 Taste and smell Importance of smell: Commonly accepted that influence how we act and fell Can stimulate memorization Can enhance sense of presence by recalling experience and modifying emotional state. Taste: Useful as interaction modality? Unexplored research area. Important role in perceiving the world Smelling Interface Contains different odorants and a system to deliver them through air and a control algorithm to determine the mix of odorants, its concentration and the time of the stimulus. Smelling Screen (Matsukura, Yoneda, & Ishida, 2013) delivers odorants through a four fans system in arbitrary positions of the screen. 94 95 Taste Interfaces only marginally addressed and few taste interfaces can be found in literature. Iwata, Yano, Uemura, & Moriya, 2004 Food simulator addresses chewing simulation releasing flavoring chemicals resistance to the mouth Playing sound Taste is very difficult to display because it is multimodal sensation composed of chemical substance, haptics and sound 96 VR Frameworks Which framework is more suitable? How to interact/use the tool Documentation Availability Extensibility Which Modules/Interfaces it provides Users Graphic Engines OpenSceneGraph OpenSG VTK 14
invrs Institute of Graphics and Parallel Programming Johannes Kepler University, Linz, Austria (2006) C++ OpenSG as a scene graph engine Network distributed virtual world Pre-defined navigation and interaction techniques Configurable via XML Mostly used on academic environments Austrian/German Universities 98 invrs Source: http://dev.invrs.org/projects/invrs/wiki 99 invrs Applications 100 VR Juggler Iowa Center for Emerging Manufacturing Technology, Iowa State University, Iowa, United States (1998) C++ Supports many graphics Engines OSG, OpenSG, OpenGL, VTK Huge variety of modules Network distribution through NetJuggler Extended over time Ex.: VR JuggLua Used for many different purposes Commercial / Academic Mina do Lousal (PT) Korea Ocean Research and Development Institute Configurable via VRJConf Backend XML Entertainment - Space Trash Source: http://www.invrs.org/app_gallery.php Safety Training - SAVE Walkthrough VR Juggler 102 VR Juggler Source: http://vrjuggler.org/features.php 15
104 Vizard WorldViz, Santa Barbara - CA, United States Pyton Scripts + GUI OpenSceneGraph - Possible to expand vizard funcionalities Support many comercial Devices Physics Engine Research & Science (VHIL - Stanford U), Architecture & Construction (Archidimex - Netherlands) Vizard Development Edition Vizard Development Academic - Single License $ 3,990 Vizard Development Academic - SITE License 5 Seats $ 6,990 Vizard Enterprise Edition Vizard Enterprise Academic - Single License $ 5,990 Vizard Enterprise Academic - SITE License 5 Seats $ 9,990 Vizard Plug-Ins Vizard ARToolworks Add-on Development Edition $ 570 Source: Korea Ocean Research and Development Institute Vizard ARToolworks Add-on Enterprise Edition $ 7,990 Source: http://vrjuggler.org/, Last Access Dec 03 2013 Vizard 106 3DVIA Studio Dassault Systèmes, Vélizy-Villacoublay, FRANCE Former Virtools Scripting Language + LUA Closed Engine Many GUI Tools to work on models, animations and scenes Price around 9000 USD + 3000 USD/yr Development Environment Applications Source: http://www.3ds.com/fileadmin/products/3dvia/3dviavirtools/demoshowcase/html/browse.html?br=1 Source: http://www.worldviz.com/products/vizard 107 Other Frameworks 108 Conclusion 109 VRPN FreeVR Complex Architecture Many Abstractions Not trivial to Install / Configure Open Still on Beta (last version Dec/2011) LVRL (PUC-Rio) Not Released Yet Avango NG DIVERSE Hardware manipulation can be hidden Implementation point of view To choose one - What is the problem? ViRAL EON Studio 16
Interaction with 3D world Main tasks in 3D world can be subdivided as: Navigation or Travel Selection Manipulation Navigation Three sub-tasks: Direction or Target selection: how to move or where to move Velocity/acceleration: speed control Input conditions: how travel is initiated, continued and terminated. 110 111 Navigation Navigation (Bowman & Hodges, 1999) 112 (Bowman et al, 2004) 113 Selection Three sub-task: Indication of object Selection of object feedback Selection (Bowman et al, 2004) 114 115 17
Manipulation Main bibliography - G. Burdea and P. Coiffet, Virtual Reality Technology, 2 nd ed. Jonh Wiley and Sons, 2003 - Craig, A., Sherman, W., Will, J., Developing Virtual Reality Applications: Foundations of Effective Design, Morgan Kaufmann, 2009 - Gutiérrez, M. A. A., Vexo, F., & Thalmann, D. (2008). Stepping into Virtual Reality (p. 214). London: Springer London. - Bowman, Kruijff, E., LaViola, J., & Poupyrev, I. (2004). 3D User Interfaces: Theory and Practice (p. 512). Addison-Wesley Professional - J. Vince, Introduction to Virtual Reality, Springer, 2004 (Bowman et al, 2004) 116 117 Aknowledgement The authors of these slides is greteful to: Burdea and Coiffet for making available the slides supporting their book All colleagues and students that have contributed in any way 118 18