Overview. From regular images to Panoramas. Advanced Media. Navigable images A single-image immersive environment

Transcription

1 University Research Chair Ambient Interactive Media & Communications Director: Multimedia Communications research Laboratory (MCRLab) and ICT-Cluster of Ontario Research Network on E-Commerce (ORNEC) Abdulmotaleb El Saddik Haptics Technologies: Theory, Applications & Challenges Overview Mixed Reality The sense of Touch Haptics Devices Haptics Applications Collaborative Hapto-Audio-Visual Framework Haptic Application Meta Language (HAML) Haptic Playback Haptic authoring Case studies Conclusion & Challenges May Advanced Media Conventional media: audio, video, text, image, Emerging types of media provide greater interaction. Wearable/enhanced interfaces Head-Mounted Displays 3D Goggles Gloves Haptics (e-touch) Others The high-level of immersion and interaction with these new devices and media can lead to unprecedented rich learning experience. From regular images to Panoramas Navigable images A single-image immersive environment Very effective for learning Historic site Museums History Coliseum 360 degrees May May

2 Context Sensitive Panoramas Panoramas that show additional information depending on the context. User can focus on a specific learning object to get information about it. There are many ways this focus can happen: click binary square markers gaze recognition gesture recognition Context Sensitive Panoramas ( ) May May Context Sensitive Panoramas ( ) Virtual Worlds 3D Virtual Environments; Virtual Reality based worlds The idea is to give the impression to the user that what is being experienced is virtually real. We haven t reached that yet, but in some cases we can offer a somewhat believable experience. Some believe this can be as simple as having a 3D model on the monitor, made in VRML for example. The experience is the key: without it it s not real. For this experience, we need additional equipment: HMD Stereo Goggles Force-feedback tools May May

3 3D Worlds avatars 3D Model Can be created in two ways: Programming the model, usually with an authoring tool. 3D scanning of an actual object/scenario. The resulting 3D model can be stored in a standard format, such as the Virtual Reality Modeling Language (VRML) or X3D. May May D Capturing Caves Similar to scanning normal 2D images, one can also scan an object to create a 3D model of it. Medical imaging (MRI, X-Rays, Ultrasounds) 3D polygonal animation (Maya, 3D Studio) Computer simulations (weather, biological) May May

4 Caves ( ) Simulator May May D Video 3D Video System stereo view of a video, with or without free viewpoint free viewpoint: similar to the concept of panoramas as applied to images, but applied to videos. On its own or merged with synthetic 3D environment Potentially very large data sets May May

5 3D Video Capturing Multiple spatially calibrated video cameras using image processing techniques Personal note: we re currently working on an e- Learning project with Zaxel involving Piano Pedagogy. Mixed Reality A definition from Wikipedia: Mixed Reality is the merging of real world and virtual worlds to produce a new environment where physical and digital objects can co-exist and interact. Zaxel Virtual Viewpoint system in the DISCOVER Laboratory May May Paul Milgram and Fumio Kishino s Taxonomy Image Illustrated RV Continuum In December 1994, P. Milgram and F. Kishino published A taxonomy of mixed reality visual displays in IEICE Transactions on Information Systems, Vol. E77-D. Mixed Reality (MR) consists of two parts Augmented Reality (AR) and Augmented Virtuality (AV). Source: T. Sielhorst, Technische University Munchen May May

6 Mixed Reality vs. Virtual Reality Similarity Use computer generated virtual data Can be interacted by user Difference Mixed Reality Real world + virtual world Enhanced reality to interact with user Virtual Reality Computer generated 3D environment Totally virtual world, nothing from real world User is immersed May Augmented Reality vs. Augmented Virtuality Similarity Utilize computer generated data Merge objects from real world and virtual world Difference AR Augment the real world with synthetic digital data, which means overlay virtual objects in the real world scene Enhance user sensation of the real world AV Augment the virtual environment with data from the real world, which means overlay real objects in the virtual world scene Enhance user sensation of the computer generated virtual world May Application Medical Liver Surgery Planning System Application Industrial Assembly, Repair and Maintenance A project of ICG (Graz University of Technology) AR system includes Stereoscopic see-through HMD Infrared tracking system Tracked input devices Rending & tracking workstations Purpose Support radiologist to prepare diagnostic data Give surgeons precise information for optimized decision making Display expected result (above) Show assembly instructions (right) Source: May May

7 Application Tour Guide Use customized message to replace commercial sign Source: Display special information on site Application Education AR Volcano Kiosk A HIT Lab project Purpose Demonstrate AR technology Teach people about volcanoes in more attractive way May May Ambient Intelligence I want to feel the handshake? Unobtrusive hardware - dynamic and massively distributed miniaturisation, nano-technology, smart devices, sensors etc.) A seamless mobile/fixed web-based communication infrastructure interoperability, wired and wireless networks etc. Natural feeling human interfaces intelligent agents, multi-modal interfaces, models of context awareness etc. Dependability and security self-testing and self repairing software, privacy ensuring technology etc. May May

8 BMW s idrive Bring the Touch - HAPTICS CyberGlove Science of applying force feedback and tactile sensation to human interface with computers. May CyberGrasp Exoskeleton CyberForce Tactile Feedback System VirtualHand for MotionBuilder Fact is that We rely on our sense of touch to do every day tasks such as: Dialing a touch-tone phone Finding first gear in a manual transmission car Playing a musical instrument like a guitar or a piano We heavily rely on the tactile and kinesthetic cues we receive. Tactile cues include: textures, vibrations, and bumps; Kinesthetic cues include those such as the weigh of a stone and the impact of hitting a tennis ball. May What is Haptics? Haptics: refers to Touch Derived from the Greek verb haptesthai Refers to sensing and manipulation through touch Comprises both Tactile and Kinesthetic senses Haptics: An emerging interdisciplinary field for Understanding the human haptic sensory and motor characteristics Developing a computer-controlled system to allow us to physically interact with our environment via touch May Brief History of Haptics Early 20th century The term began to be used in experimental psychology Study human touch perception 1950s 1980s Force feedback for Tele-operation in Robotics Sensory design and processing 1990s 1995s The term computer haptics was introduced Break-through in haptics: PHANToM 1995s currently World wide active research: perception, devices, rendering, modeling, etc. May

9 Bidirectional exchange of energy Principle of Operation Passive Devices Programmable dissipation; f (time or position) Visual/Audio Computer Impedance Control The actuators act as force source, and position is measured Active Devices The energy exchange is entirely a function of the feedback control User Gesture Touch Haptic interface Admittance Control The actuators act as position source, and then the force is measured May May Haptic Interfaces Haptic Rendering A haptic interface is a device which allows a user to interact with a computer by receiving tactile/force feedback. A haptic device achieves the tactile feedback by applying a degree of opposing force to the user along the x, y, and z axes. A haptic interface serves to orient users to the location and nature of objects in a virtual space. May Source: SensAble Tech., USA Haptic rendering is the process of computing and generating forces in response to user interactions with virtual objects. Haptic rendering enables a user to touch, feel, and manipulate virtual objects through a haptic interface. May Source: SensAble Tech., USA 9

10 Graphical/Haptic Rendering Principles of Haptic Rendering A haptic rendering algorithm is made of two parts: Collision Detection: As the user manipulates the probe of the haptic device, the new position and orientation of the haptic probe are acquired, collisions with the virtual objects are detected Collision Response If a collision is detected, the interaction forces are computed using preprogrammed rules for collision response, and conveyed to the user through the haptic device to provide him/her with the tactual representation of 3D objects and their surface details. Source: SensAble Tech., USA May May A Problem in Haptic Rendering Haptic Interpolation Haptic technology is difficult to achieve convincingly because the human sense of touch is far more sensitive than the senses of sight. A picture needs to be refreshed at 30 frames per second to trick the eye into thinking it is continuous. However, the sense of touch requires that a sensation be updated 1000 times per second or more to relay a convincing tactile experience. It is almost impossible to simulate it at such a high frequency. To display haptics at 1000Hz or higher, we will interpolate the haptics between two simulated states using the necessary high frequency. How could we solve the problem? May May

11 Tactile Displays Tactile Displays Render feedback data that presents an object s surface geometry or texture and enable the user to feel the surface of the virtual objects Tactile sensation can be applied in three ways: Vibration: Enable the user to feel the texture of the surface by using electrical vibrators CyberTouch [Immersion Corporation]. Small-scale shape (Shape display): Convey information about the shape and surface texture of an object Consist of an array of closely-spaced pins that can be individually raised and lowered against the finger tip to approximate the desired shape Thermal display May May Courtesy of University of Exeter Kinesthetic Interfaces Low degree of Freedom DOF: (Degree of freedom) is the number of parameters which may be independently varied Low DOF Devices: (1 to 3 DOF) Types of 1 DOF interactions include opening a door with a knob Examples of 2-DOF exist in everyday life-using a mouse to interact with a PC 3-DOF interaction, the force direction isn t trivial High DOF Devices: (4 to 6 DOF) Very High DOF; (More than 6) Other examples come from devices that have been developed for the gaming industry such as haptic steering wheels and joysticks and games pads The Pantograph is typical example of two actuated degrees of freedom in the horizontal plane May May

12 Low degree of Freedom High Degree of Freedom The haptic master is a commercial example of a 3 DOF force controlled haptic interface The Phantom device can exert forces at one point in three dimensions 6-DOF Tactile Simulator A six-degree-of-freedom hand controller with force feedback capabilities designed over a mobile platform [4] 6-DOF DELTA It offers 6 active degrees-of-freedom in translation and rotation and was designed to display high-fidelity, high-quality kinesthetic and tactile information [5] May May Very High Degree of Freedom Commercial Haptic Products The Mechanical Design of a Haptic Interface for the Hand, from the Scuola Superiore S. Anna The PERCRO Laboratory is only capable of actuating the index finger and thumb with CyberGrasp, its force-reflecting exoskeleton fits over a CyberGlove and adds resistive force feedback to each finger The Rutgers Master II - New Design Force-Feedback Glove from the Rutgers University is able to provide force feedback of 16 N to each of the fingers Haptic Knob BMW idrive [Immersion Corporation] Vibetonz Mobile Player [Immersion Corporation] May Logitech Wingman Force Feedback Mouse May 2007 [Logitech] 48 Laparoscopic Surgical Workstation [Immersion Corporation] 12

13 Haptic Gloves Examples Haptic devices Ah, pictures at last! CyberForce [Immersion Corporation] CyberTouch [Immersion Corporation] May CyberGlove [Immersion Corporation] May Examples..and more pictures! May And many many more Haptic Applications Medicine Visually impaired Rehabilitation Tele-Surgery Education and Training Entertainment & Games Scientific Data Visualization E-commerce Arts and design May

14 Blind reading from computer Pin pad: using the device of Shape display Entertainment 3-D computer simulation Courtesy of University of Exeter newton.ex.ac.uk/medphys/pages/array1.html May May From Swiss Federal Institute of Technology Website Education and tele-training Surgery Training One example can potentially include a medical training system using a simulator and virtual reality Where a haptic system provides doctors with the feel of virtual patients, in this example, an abdominal aortic aneurysm surgery Virtual Reality for Surgical Training May May

15 Military Training Driving school Intensively using haptic devices Bomb squad similar to the surgery training Air force training AH-64 Apache attack helicopter training Armed tank training M1A1 training May May Collaborative Hapto-Audio- Visual System Current State of CHAVE Development Video Audio Haptic Device Visual System Haptic Device Sensor Actuator Audio/Visual Rendering Haptic System Haptic Rendering May Simulation Engine Network Interface Module Distributed Collaborative Virtual Environment Diversity of hardware systems FCS Control Systems: HapticMASTER Immersion: CyberForce, CyberGrasp, CyberTouch, CyberGlove SensAble: Phantom Omni, Phantom Desktop, Phantom Premium Among others Diversity of software Novint: e-touch API, e-touch Sono, etc. Reachin: GeoEditor, Laparoscopic, Trainer 2.0 SensAble: OpenHaptics Toolkit, GHOST SDK 4.0 Immersion: various SDKs GAMMA: dab, ArtNova Among others May

16 Current State of CHAVE Development The problem s Diversity of applications Entertainment, education and training, medicine, Programming language C, C++ Java Haptic Application Haptic Device Haptic API Virtual Environment Device heterogeneity Use heterogeneous devices in the same application API heterogeneity Usually the APIs are associated with devices OpenHaptics e-touch API, GHOST SDK CHAI 3D Library May VRML X3D Java 3D, Xj3D OpenGL CyberGarage CyberX3D No standard assembly line development Device-specific application development environments May AmI-based Haptic Framework C++ FRICTION ROUGHNESS MOVEMENTS DESCRIPTION (x, y, z) SOFTWARE May INTELLIGENT AGENT PATTERN RECOGNITION PATTERN OF THE MOTION BEHAVIORAL DATA REPOSITORY STRENGTH APPLICATION Human Patterns Re-habilitation Puzzle Games OpenGL VRML Phyton API SINGLE CONTACT POINT FORCE FEEDBACK SPEED OF MOTION ( v ) Reachin MULTI POINT FORCE FEEDBACK HARDWARE PHANTOM VirtualHand ELASTICITY COLLISION DETECTION FINGER DEXTERITY Haptic Application Meta-Language HAML Enables Plug and Play! Application data Application name and type Interaction type Haptic interface data Identification: Device ID, serial number, manufacturer name Characteristics: control type, DOF, position and force resolution Haptic rendering, Meta metadata, and Copyright User Profile Application Requirements AmI Layer Authoring May HAML Repository Engine Application Description MPEG-7 Audio/Video Augmenter Application Factory 16

17 HAML Structure HAML HAML Schema Overview - 1 Application Description General application information: application name, type. Interaction task: navigation, selection, and manipulation. Interaction techniques: moving relative to object, moving the object, possessing the object, etc. System requirements: Computer specifications, device/sdk/api/driver support Meta metadata: describe the HAML document instance. May May HAML Schema Overview - 2 Haptic Device Description Observation Characteristics Physical properties Quality characteristics Response characteristics Spatial/Temporal characteristics Device geometry Temporal characteristics Description and documentation Device information and identification Device reference History and system requirements May HAML Schema Overview - 3 Haptic API Description API general information API name, version, programming language. API generic functions Device calibration, initialization, object creation and deletion Scene graph loading, force interaction Error reporting and handling Function error, force error, device error, rendering error Haptic and graphic scene synchronization Synchronous or asynchronous callbacks May

18 HAML Schema Overview - 4 Haptic Rendering Description Haptic rendering system Haptic rendering API and servo loop rate Collision detection Collision detection approach Collision detection techniques Types of queries Response schemes Force response Force computation technique Response technique Control algorithms Force generation method Interaction models May HAML Schema Overview - 5 Graphic Rendering Description Graphic rendering system: Graphic rendering API name, release version, graphic rendering refresh rate, Contact model Historic or non-historic Avatar representation: can be point-based, multi-point based, or ray-based Object modeling methodology: polygonal, bi-cubic parametric patches, Constructive Solid Geometry (CSG) Model type: rigid, deformable, or dynamic. May HAML Schema Overview - 6 Quality of Experience Description Haptic perception Haptic stability Quality of Service (QoS) parameters Haptic Interface quality HAML Schema Overview - 7 Haptic Data Description Data unit Format, range, resolution, type Data acquisition Sampling technique Encoding technique Data compression Compression algorithm Compression parameters May May

19 Implementation MPEG 7 Standard that allows interoperable search and access to multimedia Descriptions are defined in the form of Descriptors (D) and Description Schemes (DS) XML-based Schema Instantiate XML Descriptions (DS) Omni Phantom Device The Virtual Maze Application Excerpt of the Omni DS <DescriptionDocumentation> <Identification> <DeviceName>Omni</DeviceName> <DeviceType>PHANToM</DeviceType> <Model>Omni</Model> <SerialNumber> </SerialNumber> <Manufacturer>SensAble Inc. </Manufacturer> </Identification> <OverallInformation> <Driver>Phantom Device Driver V </Driver> <Platform>Intel-based PCs</Platform> <CompatibleAPI>OpenHaptics CHAI 3D Reachin </CompatibleAPI> <Owner>University of Ottawa</Owner> <Operator>DISCOVER Lab</Operator> </OverallInformation> <DeviceReference>Ground-Based</DeviceReference> </DescriptionDocumentation> May May HAML-Framework Haptic Playback The ability of displaying prerecorded haptic information using a haptic interface Haptic information can be: Spatial trajectory Applied forces Both (not possible with current technologies) May May

20 HAML-based Player System Haptic Recorder Capable of retrieving, filtering, and storing compacted sets of data in a local database Haptic Data Repository HAML Repository Uses three algorithms: Intelligent filtering algorithm Thread-management algorithm Input/Output optimization algorithm Create new ManagementThread(); StartManagementThread(); While(Simulation is Active) { RetrieveDataFromHapticDevice(); FilterRetrivedData(); SaveFilteredDataInMemory(); if(savetodisk Flag is Triggered) { SaveToDisk() } } May May The HAML Feature Extractor The Haptic Player (Video) HAML is an XML-based description language for haptic simulation. Generates standardized HAML descriptions for the captured data. HAML file includes: Repository location The haptic interface Force computation method Measurement units Workspace reference system May <Data Type = "Database"> <DataFormat> MSACCESS </DataFormat> <DBSchema>Haptic_Session</DBSchema> <DBFile> C:\\Users\\ test.mdb </DBFile> <Data Structure> <Item Name="TimeStamp" Type="int" Unit="ms"/> <Item Name="force" Type="float" Unit="N"/> <Item Name="position" Type="float" Unit="N"/> <Item Name="velocity" Type="float" Unit="m/s"/> </Data Structure> </Data> <Force_Computation> <Name>Spring-Damper</Name> <Stiffness> 0.5 N/cm</Stiffness> <Damper> 0.5 Nsec/cm</Damper> </ Force_Computation > <Stimulus> <Name>Sample Recording</Name> <Workspace> x x cm</workspace> <Center> x x cm</center> <Sampling_rate>1000</Sampling_rate> <Orientation>0x0x0</Orientation> <Duration>343535</Duration> </Stimulus> <Device> <Name>Phantom Omni</Name> Haptic Player: Makes the transformation to the player device frame based on HAML descriptions Provides extrapolation and interpolation of haptic data Provides workspace scaling Comprises three components: The HAML loader The transformation component The haptic rendering component May

21 The Haptic Player - Transformation The Haptic Player - Haptic Rendering The transformation between the two frames Transformation matrix K xxv ( θ) + C( θ) R = K xyv ( θ ) + K zs( θ) K xzv ( θ) K yc( θ) A A T = [ B R, PB ] ORG A B May K V ( θ) K S( θ ) xy K V ( θ) + C( θ) K V ( θ) + K S( θ) V( θ) = (1 Cos( θ)) K xx K x. K x S ( θ ) = Sin( θ ) C ( θ ) = Cos( θ ) yz yy z x K xzv ( θ) + K ys( θ ) K yzv ( θ) K xs( θ) K V ( θ) + C( θ) zz = K xy = K x. K y Computes the driving force (F) using equation Playback approach F K p X + K d ν = X = X n X c //move device to playback starting point Xs = GetPlaybackStartupPosition(Database); Do { SendForceToDevice(Fs); //Fs is a step force Xd = GetCurrentPosition (Device); e = Xs Xd; } while (e < K) // K is a minimum error //start the playback while (Xn = GetNewPosition(DataBase)) { Vn = GetNewVelocity(DataBase); Xd = GetCurrentPosition (Device); F = Kp*(Xn Xd) + Kd*Vn; SendForceToDevice(F); } May 2007 //Kp and Kd are stiffness 82 and damping factors Performance Evaluation Three haptic devices were used: Phantom Omni MPB Freedom 6S CyberForce system Two tests: Playback quality (same device) Device independence Performance Evaluation Player Quality Quality measures Mean-Square Error (MSE) i MSE = MSE standard deviation Test trajectories n = ( X X ) + ( Y Y ) + ( Z Z ) Five Latin vowels (a, e, o, u, and i) Two numbers, one (1) and five (5) ip ir n ip ir ip ir 2 May May

22 Performance Evaluation Player Quality Handwriting Learning System 9 MSE and standard deviation (mm) MSE (mm) a e o u i Shapes May May Haptic Applications Authoring Tool HAMLAT System Architecture Building haptic applications on the fly HAMLAT Authoring Tool HAML Player Two parts of the system: The developer interface Create meshes and assign haptic properties Export in HAML format The application player By utilizing the HAML file, they build the haptic simulation. It has a full fledged haptic rendering mechanism. Modeler HAMLAT Editor Graphic Haptic Editor Editor HAML Engine HAML Repository HAML Composition HAML Schema Rendering Engine Haptic Haptic Rendering API Graphic Graphic Rendering API Application HAML File HAML Repository HAML Renderer Haptic Graphic Rendering Rendering Haptic Graphic API API End User May May

23 HAML Description HAMLAT Authoring Tool HAML structure is divided into four description schemes (DS): The application The author The system The scene <?xml version="1.0"?> <HAML> <ApplicationDS> </ApplicationDS> <AuthorDS> </AuthorDS> <SystemDS> </SystemDS> <SceneDS> <Object> <Type> </Type> <Name> </Name> <Location> </Location> <Rotation> </Rotation> <Geometry> <VertexList> <Vertex> </Vertex> </VertexList> <FaceList> <Face> </Face> </FaceList> </Geometry> <Appearance> <Material> </Material> </Appearance> <Tactile> <Stiffness> </Stiffness> <Damping> </Damping> <SFriction> </SFriction> <DFriction> </DFriction> </Tactile> </Object> </SceneDS> </HAML> HAMLAT Editor: a GUI that enables developers to: Create/import virtual objects Enable/disable haptic interaction Assign haptic properties to the selected object(s) HAML Engine Responsible for generating a HAML file that fully describes the application Rendering Engine Renders the graphics and haptics of an application by utilizing HAML description file May May HAMLAT Implementation (Video) Blender: Open source Full-fledged 3D graphical renderer Physics and game engine Adaptive user-interface Plug-ins incorporated in Blender source code Haptic panel in Blender GUI Haptic rendering extension Implemented in C++ using OpenHaptics and OpenGL HAMLAT Implementation Two rendering modes Graphic rendering: using Blender s rendering Haptic rendering: using our haptic Plug-in HAML Import/Export Import HAML objects/scenes Export authored environment as HAML file May May

24 HAMLAT Implementation HAML Player Acts as a player for haptic applications authored using HAMLAT Select haptic device to use Save application status Save interaction data May May