Multi-modal Exploration of Large Scientific Data Using Virtual Reality

Transcription

1 1 Department of Computing Science University of Alberta Multi-modal Exploration of Large Scientific Data Using Virtual Reality Dr. Pierre Boulanger Department of Computing Science University of Alberta Shuttle Radar Topography Mission (SRTM) February 11, 2000, the Shuttle Radar Topography Mission (SRTM) was launched into space as part of one of the payload of the Shuttle Endeavor. Using a new radar sweeping technique most of the Earth's surfaces was digitized in 3D in approximately 10 days. SRTM acquired enough data during its mission to obtain a near-global highresolution database of the Earth's topography. Terrain Model of Mount St. Helens Terrain Model After Compression and Hole Filling Low Altitude Airflow Over Mount St. Helens Terrain Model Rendering Aburra Valley Colombia CFD Model Convective Winds Simulations Based on Landsat IR Data

2 2 Virtual Analysis of a Francis Turbine at La Herradura in Colombia The main objective of the DIFRANCI Project is to apply a condition assessment methodology following holistic approach to the maintenance of the Francis turbines of "La Herradura" hydropower plant in Colombia. Project in collaboration between: Empresas Públicas de Medellín Colombian Agency for Science and Technology EAFIT University, Medellin, Colombia EPFL, Lausanne, Swizerland UofA, Edmonton, Canada Digitizing the Turbine Using a Hand Held Scanner Scanning Using Handy Scan from Creaform 3D Final Scanned Reversed Engineered Model Scanned Model of the Turbine Extracted FEM Mesh Rapid Virtual Prototyping Once a 3D model is created, virtual prototyping allows product testing without the need to build a real prototype Allows for shape and functional optimization Allows to tract the complete life cycle of a product Rapid Virtual Prototyping requires powerful computing infrastructure especially if it is interactive CFD Simulation of Francis Turbine Project Particle Flow

3 3 Pressure Variations vs Time Results of CFD Analysis Wall Pressure Comparisons of the computed pressure recovery coefficient with the experimental values, medium size mesh, 1.15ψ= in the case of the FLINDT draft tube. The UofA/EAFIT Virtual Wind Tunnel Definition of Interactive CFD Need: A set of tools that allow the designer to test interactively the behavior of a design under various flow conditions. Task: The main task performed by an interactive CFD system can be defined as solving simulations of fluid flow for an object, given the possible scenarios defined by the user and to display the results interactively. User/Consumer: The target market is none other than designers interested in testing their design s behavior under fluid flow conditions to optimize and test their design. Interactive CFD User needs UofA/EAFIT Virtual Wind Tunnel Architecture

4 4 Standard Scientific Visualization/Simulation Pipeline Remote Simulation Parameters Desktop Data Simulator Filter Map Render Data Repository High Speed Network Image Problems With Current Practice Hard and expensive process to determine dominant parameters in a CFD simulation model. Simulation runs cannot be steered resulting in useless computations. It is like working blind. Current pipeline do not allow collaboration. It is always after the fact that the simulation data is analyzed and shared by a group of engineer. Even Better: Collaborative Visualization/Simulation Steering Environment Simulation Parameters User1 UofA Advanced Collaborative Immersive Environments New VizRoom 3D Graphic Rendering Massive Storage Data Client 1 Filter Map Render Image Haptic Rendering Simulation Server User n 3D Sound Rendering AMMI Lab Local High Speed Network Data Client n Filter Map Render Image Input Sensors Definition of Virtual Reality A virtual reality system is an interface between a human and a machine capable of creating a real-time sensory experience of real and artificial worlds through the various human sensory channels. These sensory channels for man are: Vision, Audition, Touch, Smell, and Taste. Definition of Real-Time Real-time, in virtual reality, means that the computer system can detect the input of the user and react to it fast enough so that it appears to be instantaneous. Burdea, 1993

5 5 How is VR Different than CG? Objects in the environment have a strong sense of spatial presence, creating the effect that the objects exist independently of the user. Control of interaction within the environment is often through direct manipulation of objects as in the real world. How is VR Different than CG? The computer interface is hidden in the sense that the user interacts with objects in the environment rather than a computer which controls objects in the environment. The user is immersed in the environment, i.e., the user experiences the environment from within. Immersion= Presence Presence is a state of consciousness where the human actor has a sense of being in the location specified by the displays. The unique feature of "virtual reality" systems is that they are general purpose presence transforming machines. Multimodal vs. Multimedia Multi-modal systems use more than one sense or mode of interaction e.g. visual and aural senses: a text processor may speak the words as well as echoing them to the screen Multi-media systems use more than one media to communicate information e.g. a computer-based teaching system - may use video, animation, text and still images - different media all using visual mode of interaction - may also use sounds, both speech and non-speech Multimodal Human Computer Interaction Human Computer Multimedia output Information processing Interface Information processing Internal perception / action feedback loop Multimodal input AI agents internal decision loop Level of Interactivity Reactive: where little user control takes place over the content's structure with program directed options and feedback Co-active: providing user control for sequence, timings and style. Pro-active: where the user controls both structure and content at most of the levels.

6 6 First Level of Function Segregation Virtual Wind Tunnel Interaction Map Data Exploration and the Mapping Problem Visualization n-d Data Mapping Virtual World Visual Haptic Sound Westgrid 4K x 2K Display Visualization Toolkits Computation/Analysis + Visualization NIH Image and its PC version Scion Matlab Programming Toolkits The Visualization ToolKit (VTK) Insight ToolKit (ITK) VisAD and Vis5D (also visualization spreadsheet ) SCIRun Graphical Programming Toolkits Open Data Explorer (OpenDX) Paraview Advanced Visual Systems (AVS/Express) Amira Slicer Multivariate Display Techniques Glyphs: 2D Scalar Heterogeneous Techniques: 2D and 3D Texture Spot Noise (van Wijk), Healey, Ware Layering 2D Scalar: Slivers, DDS (3D?) 2D Scalar, Vector, Tensor: Laidlaw, Crawfis Problem Reduction Dimensional reduction, Cluster analysis Smart Particles Time and space multiplexing Mapping different fields over time Magic

7 7 Glyph: Flow Probe Multiple Views in Space Definition of Haptic Gibson (1966) A haptic system is defined as "The sensibility of the individual to the world adjacent to his body by use of his body". The haptic perceptual system is unusual in that it can include the sensory receptors from the whole body and is closely linked to the movement of the body so can have a direct effect on the world being perceived. Human Haptics Two complementary channels: ~ Tactile Strictly responsible for the variation of the cutaneous stimuli Presents spatial distribution of forces ~ Kinesthetic (Proprioception) Refers to the human perception of one s own body position and motion Presents only the net force information Tactile Display CyberTouch of Immersion Skin sensation is essential for many manipulation and exploration tasks, for example, medical palpation. Tactile display devices stimulate the skin to generate these sensations of contact. The skin responds to several distributed physical quantities: 1. High-frequency vibrations: Surface texture, slip, impact, and puncture. 2. Small-scale shape or pressure distribution 3. Thermal properties CyberTouch is a tactile feedback option for Immersion's CyberGlove. It features small vibrotactile stimulators on each finger and the palm of the CyberGlove. Each stimulator can be individually programmed to vary the strength of touch sensation. The array of stimulators can generate simple sensations such as pulses or sustained vibration, and they can be used in combination to produce complex tactile feedback patterns.

8 8 Other Tactile Displays Thermal properties: We infer material composition and temperature difference. Thermal display devices usually based on Peltier thermoelectric coolers. Many other tactile display modalities: electro-rheological devices for conveying compliance, electro-cutaneous stimulators, ultrasonic friction displays, and rotating disks for creating slip sensations. Haptic Rendering of Surfaces The Models of the Probe State of the Art Difficult to simulate proprioception on the entire body a point Fingertip Wrist a 3D object a line segment Arm + 2 fingers Foot VR Architecture of CoRSAIRe/CFD Environment at LIMSI CFD Exploration Using LIMSI CoRSAIRe System Menelas 2010

9 9 CFD Exploration Using CoRSAIRe System Haptic Data Rendering-I Pao and Lawrence, 1998 Tourque Nulling Transverse Damping Haptic Data Rendering-II AMMI Lab Multi-Modal Interface for CFD (Visual and Sound) Pao and Lawrence, 1998 van Reimersdahl et al., 2003 Relative Drag Feature Shift Type of Modalities Multi-Modal Exploration of CFD Flow Modalities used for the interface Visual Mono/Stereo/CAVE Haptic Perception of fluids flow Objects manipulations Setting of boundary conditions Sonification of fluids

10 10 Project Background Multi-Modal Rendering System Structure Input: Fluid field with velocity vector, pressure, and other data Changes with time Output: Sound characterizing the given fluid field Ambient: global to the whole field Local: at the point or area of interaction Local region: particles of the specific subset area around the pointer contribute to the sound Max/MSP Program Main Program as Max/MSP object Sound Solution Data Server Haptic Program Haptic Device Visualization Program Image Haptic Rendering Read from the haptic device and sends pointer info to the sound and visual programs Pointer position and orientation (converted to the data field dimensions) Buttons: interaction sphere diameter- local region Render a force feedback: Virtual walls: provides a force disallowing movement of the device outside of the data field boundary Other feedback possible: produce a force that is proportional to the flow density and its direction Visual Rendering Displays vector field, virtual pointer (microphone) and interaction sphere SGI OpenGL Performer Library for graphical representation Sound Rendering I Calculates velocity vector at the position of the virtual microphone depending on interaction sphere radius (using Schaeffer s interpolation scheme): pn( t) / rm rn s p ForallNode m( t) 2 1/ r r ForallNodes Small : from vertices of the grid cell Large: from all the vertices inside the influence sphere Velocity value & angle at that position m n 2 Sound Rendering II Two output values for both angle and velocity: Output = value / max value Output = (value / max value) 5/3 Virtual Microphone Direction Relationship between loudness level and intensity: S ~ a 3/5 [B.Gold] Thus, a function between values and amplitude should be: a = const * data value 5/3 to imply: S ~ data value v (t)

11 11 Sound Rendering III White band noise is modified in amplitude and frequency to simulate a wind effect Frequency ~ v (t) were v -> [0,1] -> [500, 1500] Amplitude ~ v * 5 / 3 were v 5/3 -> [0,1] and a 5/3 -> [0.5, 1] v (t) 5/3 Sonification Types Positive vs. Negative Amplitude Modulation velocity value is mapped to either increase or decrease in amplitude of the sound Amplitude vs. Frequency Modulation highest velocity value is mapped to either loudest noise or highest pitch noise Before vs. after interpolation many separate sounds for each vertex in the local area vs. one sound of the interpolated value at the position of virtual pointer Hypothesis According to multimodal theory adding sound rendering to visual rendering should improve exploration of CFD flow fields Testability: In our experiments, we will determine if our hypothesis is correct for an eddy localization task Simplicity: This hypothesis is simple and can easily be tested Null hypothesis: The combination of visual and auditory do not improve eddy localization efficiency. Variables Independent Variables Eddy localization in the flow field Starting point in the volume Sound mapping: frequency or amplitude modulation Sound rendering: positive or negative amplitude modulation Interpolation type: Schaeffer s or multi-sound sources Virtual microphone radius R Dependent Variables Eddy localization error Time and length of trajectory to get to the eddy Interface Evaluation Procedure 1. Design the experiment. 2. Conduct the experiment. 3. Collect the data. 4. Analyze the data. 5. Draw your conclusions & establish hypotheses 6. Redesign and do it again. Usability Experimental Setup Visual and/or Audio cues, haptic - navigation

12 12 Usability Study Experimental Setup Participants asked to locate vortex centers 40 fields: 25x25x25 Random vortex locations Red arrow / specific sound 15 warm-up trials 36 experimental trials Random start point Random setup Usability Study Results Worst results for the audio-alone system: participants are slower in locating the goal participants are less efficient in exploring the volume Participants are less precise in locating the goal Multimodal vs. Visual Only Interface Importance of Sound Rendering Parameters Specific system setup helps to improve performance Equal or better results for the multi-modal system Participants explore less space Participants are much faster in locating the goal position The Best Configuration: Positive amplitude with large radius and before interpolation Usability Study Conclusion The multimodal system is more efficient to localize eddies than either pure visual or pure audio systems Specific mapping parameters influence system performance Different audio parameters are better for audio-only than for multi-modal interface Different audio parameters might be better for different conditions Multi-modal Exploration of Electric Filds Melenas 2010

13 13 Molecule Docking Solar TErrestrial RElations Observatory (STEREO) Melenas 2010 NASA Picture May, 2006 STEREO Data Pictures by Johns-Hopkins Applied Physics Laboratory Usable Data from STEREO Stereo Pairs of the sun in visible and ultraviolet bands 3-D Reconstructions of the Corona and the sun main body 3-D distribution of plasma characteristics of solar energetic particles Local vector magnetic field Plasma characteristics of protons, alpha particles and heavy ions Trace the generation and evolution of traveling radio disturbances NASA STEREO Database STEREO Multi-Modal Interface System for Data Driven Simulations Sun Corona Simulation Sun Particle Emission Simulation Sun Electro-Magnetic Emission Simulation Multi-Modal Data Exploration Interface Earth Electro-Magnetic Disturbances Simulation

14 14 Next! The GPU Revolution SGI Petaflops in a rack True real-time simulation and visualization Next Seminar Dec 1. Recent Developments of Closed-loop Simulation and Visualization Interfaces Using GPU SGI Prism XL Tesla M2050 / M2070 GPU Computing Module