Teil 5: Aural and Haptic Displays Virtuelle Realität Wintersemester 2007/08 Prof. Bernhard Jung Overview Aural Displays Haptic Displays Further information: The Haptics Community Web Site: http://haptic.mech.northwestern.edu/ Sherman & Craik, 2002, chapter 4 1
Aural Display Paradigms stationary displays -- speakers displays that move with the head (head-coupled) -- headphones Properties of Aural Display Paradigms Aural presentation qualities Number of Display Channels Sound Stage Localization Masking Amplification 2
Aural Qualities number of display channels two sensory inputs (ears) different sounds to each ear provide cues about from where a sound is coming can display same signal to both ears (monophonic) can display different signal to each ear (stereophonic) can display different signal to multiple speakers (e.g. quadrophonc, octaphonic, surround) multiple speaker displays rely on ears to naturally localize sounds Aural Qualities Sound stage the source from which a sound appears to emanate head-referenced vs. world-referenced head-referenced sound stage moves with the head world-referenced sound stage remains fixed with the world world-referenced is how we naturally experience sound in immersive environments, sound should be world-referenced when headphones are used, head tracking is needed to reproduce a worldreferenced sound stage 3
Aural Qualities localization / spatialization localization is our brain's ability to determine the location from which a sound is emanating we localize sounds using cues such as the time and volume differences of when a sound reaches each ear, and the differences in the actual sounds caused by the shape of our bodies, especially the outer ear spatialization is our technology's ability to make a sound appear to come from particular points in space spatialization is easier with headphones because we can directly control what is heard by each ear Aural Qualities Masking loud sounds mask softer sounds physical objects can mask a sound (often without completely occluding it) speaker systems cannot effectively mask real-world sounds closed headphones are best for experiences where the participant is only supposed to hear sounds from the virtual world Amplification need to boost the sounds to hearable levels must take care not to amplify sounds too much, especially with headphones 4
Properties of Aural Display Paradigms Logistic qualities Noise Pollution User mobility Interface with tracker methods Environment requirements Associability with other sense displays Portability Throughput Encumbrance Safety Cost Logistic Qualities of Aural Displays Noise pollution (unwanted sounds) goes both ways sounds from speakers may be objectionable to others not involved systems using speakers can be disturbed by surfaces that echo User mobility Wired headphones limit mobility Speakers and wireless-headphones are okay If visual display requires wires to the head, wires of headphone can be incorporated Interface with tracker methods magnets in the speakers/headphones may interfere with electromagnetic trackers headphone magnets smaller, but closer to the tracker receiver loud sounds from speakers may interfere with ultrasonic trackers 5
Logistic Qualities of Aural Displays Environment requirements square room (like a CAVE) can be a problem for speaker displays, e.g. echoes hard-surfaced square room even worse Associability with other sense displays typically, stationary with stationary & head-based with head-based headphones easily incorporated with the visual head-based display for spatialized audio, headphones best choice even in stationary visual display Portability headphones obviously more portable than speakers not only smaller, but don't require additional amplification Logistic Qualities of Aural Displays Throughput as with visual head-based displays, headphones require time to put on / take off headphones require a separate pair for all immersed participants speakers work much better for larger groups Encumbrance speakers generally more comfortable for lengthy periods of time Safety hearing damage is possible through both speakers and headphones headphones closer to the ear though, so may be more of a concern wires to headphones are a potential tripping hazard hygiene Cost high-quality headphones less expensive than high-quality speakers but cost per listener may be less with speakers amplification system for speakers adds to the cost 6
Aural Displays - Summary Benefits of stationary displays (speakers) Works well with stationary visual displays Does not require sound processing to create a world-referenced sound stage (one that remains stable to the virtual world) More user mobility (less cables) Less encumbrance (none in fact) Faster throughput Benefits of Head-coupled displays (headphones) Works well with head-coupled visual displays Easier to implement spatialized (head-referenced) sound fields Easier to reject unwanted room noise More portable Private Spatial Audio Software e.g. principle of VRML'97 Sound node: 7
Haptic Displays Feel and Touch the Virtual Environment Two forms of haptic perception: Kinesthetic (proprioceptive, force) feedback e.g. joints angles, muscle length & tension Tactile feedback nerve sensors at skin surface (pressure, temperature, ) Morton Heilig, 1962 Simulated motorcycle ride through Brooklyn 8
Haptic Displays Primary forms of haptic displays used in VR Tactile displays End-effector displays Robotically operated shape displays Tactile Haptic Displays Teletact Glove (30 bladders) CyberTouch (vibro-tactile) Cricket prop (vibrating) Sensed by the skin -- the largest single organ of the human body Actuators mounted generally on the fingers and hand Generally no need for world grounding Bladder actuators Vibrator actuators Pin actuators Thermo-actuators 9
Tactile Haptic Displays SmartTouch electro-tactile display University of Tokio University of Karlsruhe Tactile Haptic Displays Immersion Touch Sense Tactile Feedback for Touchscreens www.immersion.com Graphical buttons can provide the familiar up and down forces of physical buttons Menu items can supply a pulse sensation when lightly touched and a confirming push-back response when pressed A rocker switch can exhibit increasing or decreasing vibrations corresponding to motor or fan speed, magnitude, or other parameter Enter, Next, and other major and minor functions can supply a consistent feel throughout an application, allowing a user to know instinctively whether they ve selected correctly Scrolling displays can provide a stop sensation when the first or last items have been reached Switch Prof. B. Jung controls can exhibit a pop effect Virtuelle Realität TU Bergakademie Freiberg Levers can offer a click response for each possible setting 10
Tactile Haptic Displays Hug Shirt a Bluetooth accessory for Java enabled mobile phones "The system is very simple: a Hug Shirt (Bluetooth with sensors and actuators), a Bluetooth java enabled mobile phone with the Hug Me java software running (it understands what the sensors are communicating), and on the other side another phone and another shirt. If you do not have a Hug Shirt but know that your friend has one you can still send them a hug creating it with the HugMe software and it will be delivered to your friend s Hug Shirt!" http://www.cutecircuit.com/ End-Effector Displays a mechanical device that provides a force to the participant's extremities generally linked to mechanical tracking sensors generally world grounded often operate with respect to a single point in the virtual world Novint Falcon (consumer market) Sensable Phantom Omni 11
End-Effector Displays 2 Phantom force feedback devices End-Effector Displays Rutgers Dexterous Master 12
End-Effector Displays Immersion CyberGrasp with force feedback via exoskeleton Immersion CyberForce CyberGrasp + 6DOF mechanical tracking http://easylink.playstream.com/immersion/cybergrasp.rm End-Effector Displays Immersion Haptic Workstation http://easylink.playstream.com/immersion/hws-3min-lq.wvx 13
End-Effector Displays - Teleoperation of Robots Sarcos Dextrous Arm Utah-MIT Dextrous Hand www.sarcos.com/teleop_videos.html End-Effector Displays Sarcos Uniport specialized device for simulation of laparoscopic surgery 14
Robotically Operated Shape Displays Only few research prototypes Use robots to place a (phyiscal) representation of the virtual world where the user is reaching. May be generic (corners and edges) May be specific (e.g. selection of switches) Usually uses a finger surrogate (sticklike object) for fast tracking, and safety. Properties of Haptic Display Paradigms Haptic presentation qualities kinesthetic cues tactile cues grounding amount of contact number of display channels degrees of freedom form fidelity resolution latency cruciality frame rate size 15
Haptic presentation qualities Kinesthetic cues nerve inputs that sense angles of joints, muscle length, tension, and resistance to muscle effort within the body. help us determine firmness, approximate shape, and physical forces 75 joints in the body (44 in the hands), so very difficult to display them all to the participant Tactile cues the sensory receptors at the skin mechanoreceptors -- shape and surface texture thermoreceptors -- heat transfer electroreceptors -- electric current flow nociceptors -- pain from tissue damage A Very Special Tactile Haptic Display Painstation - Pong arcade game - PEU Pain Execution Unit - electro shocks -heat -lashes i.e. stimulates thermoreceptors -- heat transfer electroreceptors -- electric current flow nociceptors -- pain from tissue damage www.painstation.de 16
Haptic presentation qualities Grounding force & resistance displays require an anchor world-grounded motion limited by something attached to the world e.g. Phantom, steering wheels, joystick self-grounded motion limited between two parts of the same thing self-grounded systems more portable but limited in the types of forces that can be displayed body-mounted haptic displays, e.g. tactile CyberGlove, exoskeletons Number of display channels how many points of contact with the body e.g. phantom has one point where the user can influence the virtual world phantom in each hand two channels Prof. B. Jung Virtuelle Realität TU Bergakademie Freiberg Haptic presentation qualities Degrees of freedom 6-DOF in unconstrained movement 1-DOF display: how far down a tube can you insert a laparoscope camera 2-DOF display: how far down a tube, plus twist 3-DOF display: location of the finger tip or stylus 6-DOF display: location and orientation of a finger tip or stylus 17
Haptic presentation qualities Form the shape of what the participant interacts with generic: sphere, stick, plane specific object (prop): handgun, steering wheel amorphous: able to change shape to multiple specific representations e.g. glove, pin display Fidelity safety may require low-fidelity in many circumstances how rapidly can the system change to the proper display (force, temperature, etc.) force displays can be rated by a maximum stiffness measurement 20 Newton/cm will generally be perceived as a solid, immovable wall. 40 Newton/cm is the maximum force that a human finger can exert 10 Newton/cm is the highest force used when doing fine manipulation Haptic presentation qualities Spatial resolution differs by body region finger tips can sense differences 2mm apart 30mm on the forearm 70mm on the back Therefore higher resolution required at the fingertip sensory homunculus: mapping the human somatosensory cortex Temporal resolution low frame rate in a force display causes the object to be perceived as mushy or shakey 1000 Hz is a good minimum 18
Haptic presentation qualities Latency tolerance As with frame rate, a low latency display is crucial especially for force displays Even more important when two people are trying to manipulate a single object (e.g. try to pick up an object together) Because force displays are connected by armatures, low-latency mechanical tracking is easy to integrate Size larger displays allow broader range of motion, but are generally higher-power devices (and therefore less safe). smaller displays work well for tasks such as surgery where the operators hands do not cover large spaces Properties of Haptic Display Paradigms Logistic qualities user mobility interface with tracker methods environment requirements associability with other sense displays portability throughput encumbrance safety cost 19
Logistic Qualities of Haptic Displays User mobility World-grounded displays require the user to be near the device. For applications that don't require the user to cover much territory, this is not a problem e.g. surgery (real or simulated). Self-grounded displays can move with the user, though cables might be a concern. Interface with tracker methods A responsive & accurate tracking system is required. Mechanical tracking often works for force displays because of the linkages already present to render the force. Logistic Qualities of Haptic Displays Environment requirements Large force displays often require special rooms equipped to handle hydraulic or pneumatic pressure pumps. Smaller force or tactile displays can work on the desktop, at a kiosk, or be held in the hand. Associability with other sense displays Because the user must come in physical contact with a haptic display to get the sensation, it is difficult to hide them. Occlusive head-based displays thus often are used in conjunction with haptic displays to overcome this. Another solution is to wear the force display (e.g. a force system worn on the shoulder & arm). 20
. Virtuelle Realität Logistic Qualities of Haptic Displays Portability Small tactile and small force displays can likely be easily transported. Displays requiring a hydraulic/pneumatic system often require equipment that is hard to move -- even if the device itself is self-grounded. Of course, world-grounded system that are physically mounted to the ceiling, floor or wall tend to be hard to move. Throughput Wearable devices such as tactile gloves and self-grounded force displays usually take time to put on / take off. Force displays that interact with a stylus or other finger surrogate that the user can simply grab or release allow users to switch rapidly. Encumbrance Self-grounded, exoskeleton-style devices are generally fairly encumbering. Small force displays and glove devices are less so. Most haptic displays still involve a fair amount of wires though, so even the smaller devices must contend with them. Logistic Qualities of Haptic Displays Safety Safety is a significant concern with many haptic displays, especially large force displays e.g. large robots, large exoskeletons Temperature and other tactile displays may also be hazardous. Often displays that can produce forces or other potentially harmful results are equipped with a "drop-dead" switch. Cost Most haptics displays are costly. They are still not very widely used, and thus have not reached mass market prices. Exception: force feedback steering wheels, joysticks, and vibrators in game controllers. 21
Benefits of Tactile displays helps in the fine manipulation of virtual objects (some) can be added to some end-effector displays can be world or body grounded (usually body-grounded) body-grounded method is mobile generally less-expensive generally portable Benefits of End-effector displays can be world or body grounded (exoskeletal is body-grounded) exoskeletal method is mobile world-grounded method is not very encumbering fast and accurate tracking is usually built into display Benefits of Robotically operated shape displays can provide a very realistic haptic display world-grounded display fast and accurate tracking is usually built into display works primarily with head-based visual displays Other Senses: Vestibular (Balance) Display 22
Other Senses University of Tsukuba, Japan Food Simulator displays biting force, integrated auditory and chemical sensations Other Senses www.ambx.com Philips ambx: "full sensory surround experience" ambient lighting, fans (airflow, temperature), vibrations ("wrist rumbler") 23