Haptics Technologies: Bringing Touch to Multimedia C2: Haptics Applications Outline Haptic Evolution: from Psychophysics to Multimedia Haptics for Medical Applications Surgical Simulations Stroke-based Rehabilitation Support of the Visually Impaired Tele-robotics and Tele-operation Tele-Surgery Military Applications Media Haptic Broadcasting E-Commerce Video Games Arts and Design Mobile Haptics Haptics and Virtual reality Education and Learning Haptics for Security Closing Remarks Haptic Evolution: from Psychophysics to Multimedia Experimental psychology Beginning of 20 th century Understanding human touch perception and manipulation Robotics era Seventies and eighties of last century Create devices with a dexterity inspired by human abilities Tele-operation era Late eighties extension of sensing and manipulation to a remote location Haptics Early nineties Tactile and kinaesthetic interfaces and computer haptics Multimedia Haptics 1
Haptic Evolution: from Psychophysics to Multimedia The application spectrum is quite vast, and its trend of expansion is anticipated to increase Haptics for Medical Applications Already a strong presence in the area of medical training Professionals guide students through a simulated surgical process Example: Trainee feels the surgeon performing a precision cut on a simulated human organ Statistically, surgeons have higher success rates the more often they perform a procedure Allows surgeons to practice a procedure with less fatigue and stress If geographically separated, realistic haptic rendering is near impossible! Realistic haptic models of human tissue and organs Surgical Simulations May be used to train surgeons at remote locations In the future, tele-haptic applications will be used to perform surgical procedures on patients at remote locations. e.g., Arctic regions, the space station Examples: Needle-based procedures use needles, catheters, guide-wires, and small bore instruments for teaching straightforward procedures Minimally invasive surgery (called laparoscopic surgery) Robot-assisted surgery (da Vinci Surgical System) Simulation environments Suturing video 1 Suturing video 2 2
Surgical Simulations Eye Surgery Training Simulation Virtual eye Surgical tool Phantom Haptic Device X. Shen et al., Haptic-Enabled Telementoring Surgery Simulation", IEEE Multimedia, 2008 Stroke-based Rehabilitation Involves applying forces to the injured/disabled organ (such as the finger, arm, ankle) to help it recover its strength and range of motion helps to measure performance and to tailor performance-based exercises for each patient Enables home located rehab and remote monitoring Stroke-based Rehabilitation Game-like tasks to address certain parameters of hand movement: Based on Jebsen Test for Hand Function (JTHF) Offers continuous evaluation of the patient s improvement. Example exercise: Arranging eight cubes according to a color pattern. 3
Stroke-based Rehabilitation Based on Augmented Reality (AR) technology Post-Stroke Interactive and Entertaining Rehabilitation with ReActive objects (SIERRA) Reactive Objects react with vibration to inform the subject of being interacting with virtual objects Components: A real cup A vibrotactile actuators (pager motors) An actuator controller a bluetooth module a 9V battery Features: Inexpensive light portable The actuators are arranged to cover a whole hand when grabbing. Stroke-based Rehabilitation Rehab test 1 Rehab test 2 Rehab test 3 Rehab test 4 Rehab test 5 Rehab test 6 (foot Stroke) 4
Support of the Visually Impaired Haptics enhances the perception of blind or visually impaired people by converting visual or sound information into a haptic modality Example applications: European Union GRAB project the exploration of chart data, a city map explorer, and a simple adventure game TACTICS system interpret complex scientific data Chromo-Haptic Sensor-Tactor (CHST) To feel colors VIbroTActiLe (VITAL) system Tactile feedback as a navigation tool for the visually impaired Support of the Visually Impaired Inner fabric to protect the skin How it works? Touch by hand, cell phone, mouse For hand Hand tracking using depth image For touch-enabled cell phone For mouse Virtual touch pad 5
Tele-robotics and Tele-operation Collaborative Design Designers at various locations collaborate on the same project using a HVE Haptics offers the participants additional information regarding the project Objects may be moved, grasped, assembled, disassembled Requires position sampling and collision detection for N users Geographically separated users network performance is key Tele-robotics and Tele-operation To remotely handle dangerous or radioactive substances Applications: underwater exploration, assembly and manufacturing, micro-assembly Tele-Surgery Training surgeon on a simulated human organ Reachin Display [Reachin Technologies] Tele-surgery Train surgeons at remote locations Perform surgical procedures on patients at remote locations Brain surgery 6
Military Applications Battlefield conditions, such as the presence of artillery fire or smoke, might mask sound and vision modalities Haptics become an efficient communication channel Media Started in 1959, when tactile stimulation was used to enhance the movie The Tingler by attaching vibrating devices to the theatre seats Provide higher immersion in multimedia contents two content scenarios: Authored content Viewers are able to interact with and influence the presentation of pre-recorded content Real-time content The acquisition and display of haptic content occurs in real time Haptic Broadcasting Broadcasting multimedia streams with haptic channel Examples: Home shopping environment Viewers can touch the 3D model through a haptic device A fiducial marker position and orientation are calculated Haptic Video Annotate video stream with haptic stimuli Users feel what they see and hear Haptic YouTube video Annotated with tactile feedback using XML notation Used a haptic enabled browser and an arm band device 7
E-Commerce Haptic virtual environments gives new meaning to the phrase Try before you buy Consumers could experience aspects of the product not available using current E-Commerce technology Example: How heavy is this new PDA? How soft is this sweater? haptic interfaces can answer these questions Browser plug-in or Java application offers haptic and graphical display of a 3d model of the object on a vendor website Requires that accurate models of new products can be acquired quickly and cheaply Video Games Gaming experience comprises four aspects: physical, mental, social, and emotional. Force feedback technology enhances the physical aspects of the gaming experience Greater immersive experience to gamers Haptic information is recorded and played-back in synchronization with audio-visual data BUT: Multiplayer games require high-bandwidth, low-latency networks Logitech MOMO racing wheel [Logitech] Logitech Rumblepad [Logitech] Logitech Force 3D joystick [Logitech] Gaming When applied to video games, it gives players a higher sense of immersion/realism. S. Andrews et al., HaptiCast: A physically-based 3D game with haptic feedback, in Proc. of Future Play 2006. 8
Arts and Design Arts & Culture Due to a hands-off policy, museums do not permit visitors to handle sculptures and artifacts Haptic equipped kiosks could be used to permit visitors to feel and manipulate these objects virtually Requires realistic models of museum artifacts to be acquired so that users may truly explore them Example: tracing the contours of Michelangelo s David Digital 3D model of David [The Digital Michelangelo Project] Mobile Haptics Mobile devices utilize gesture to allow easier and more intuitive user interaction Haptics would enhance such intuitivism Tactile haptic feedback is becoming common in smart phones Example: Samsung haptic mobile phone Haptic SCH-W420 Mobile Haptics The KissMe system A neck piece to display kisses Mobile device connected to neck piece via Bluetooth Users can send kiss messages to each other 9
Mobile Haptics The HugMe system 2.5D video camera A haptic Jacket (mobile device) A haptic device Force feedback device Depth camera Local person Remote person Haptics and Virtual reality Integrating haptic interactions with existing virtual environments: Linden Lab's multiuser online virtual world Second Life Education and Learning Learners learn better a sensorimotor skill task using haptic modality Example: Haptic Multimedia learning tool: Teach handwriting of alphabets (Japanese, Arabic, etc) Three modes of operation: Full guidance Partial guidance No guidance (testing) InteraSense (too) 10
Haptics for Security Haptic Biometrics More advanced handwritten signature Environment: Virtual Check Application. Visual stimuli: Virtual pen and a virtual check. Haptic stimuli: Force and frictional feedback Haptics for Security Graphical password with haptics Utilizes the physical attributes captured during human-computer interaction (pressure and velocity) Overcomes shoulder surfing attacks Continuous verification and authentication Closing Remarks The incorporation of haptics into virtual environments is still in its infancy! A wide range of human activities, such as communication, education, entertainment, commerce, and science, would forever change if we learned how to capture, manipulate, and create haptic sensory stimuli that are nearly indistinguishable from reality Business models and frameworks are needed to make haptic devices practical, inexpensive, and widely accessible (as easy as mouse) 11
ありがとう! 谢谢! متشکرم 12