Haptic Rendering and Volumetric Visualization with SenSitus Stefan Birmanns, Ph.D. Department of Molecular Biology The Scripps Research Institute 10550 N. Torrey Pines Road, Mail TPC6 La Jolla, California, 92037 Situs Modeling Workshop, San Diego, CA, Feb. 3-5, 2003
Outline Virtual Reality Haptic Rendering Haptic Rendering for interactive molecular modeling Vector quantisation Force calculation Application SenSitus Visualization of molecular structures and volume data 1 CSB Group Virtual Reality Ivan Sutherland 1965 ultimate display The ultimate display would be a room within the computer can control the existence of matter. [ ] Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. Several definitions today Virtual Reality is a computer generated simulated environment which users perceive as real with their natural senses and with which they can interact. Replace reality by virtual reality? Necessary/useful for visualization of scientific datasets? 2 CSB Group 1
Virtual Reality Stereoscopic Viewing Head Mounted Displays Shutter LCD glasses Polarized Light Head Tracking Magnetic Ultrasonic Optical 3 CSB Group Virtual Reality Head Mounted Devices (HMD) Stereoscopic viewing Measurement of head orientation Very immersive Isolation from reality can cause sickness 4 CSB Group 2
Virtual Reality Multi-Display Systems CAVE, VR workbench 5 CSB Group Virtual Reality Projection in Multi-Display Systems Virtual Object Projection to Canvas Canvas Viewer Viewing-Pyramid Projector 6 CSB Group 3
Virtual Reality SVT Visualization of Molecular Structures VDW,CPK,etc. Combination of drawing modes within one molecule Trajectory files from MD simulations Visualization of Volume Data Isosurface Volume rendering (2D/3D Textures) VR Systems Multi-display systems Standard PC and graphics workstations 7 CSB Group Virtual Reality Application 3DS PDB VTK Core VTK Util System OpenGL Operating System 8 CSB Group 4
Haptic Rendering Haptic: haptesthai (greek) to touch Haptic Rendering Create an artificial tactile sensation Haptic feedback Skin receptors Measures temperature, pressure, vibration, slip Kinesthetic feedback Receptors in muscles and tendons Measures perceived and produced force Proprioceptive feedback Joint position receptors Measures accelleration and position 9 CSB Group Haptic Rendering Receptor properties Adaptation, spatial and temporal resolution, Important research issue High temporal resolution For realistic kinesthetic feedback force updates with 500hz 1000hz frequency necessary Problem: Visual system needs only 15-30 updates per second Programs tuned to update inner loop with 15-30hz Forces difficult to calculate 10 CSB Group 5
Haptic Rendering Devices: 195x Argonne Remote Manipulator 196x GROPE (2D) 196x exoskeleton designs 199x commercial devices: Immersion SensAble Used here: SensAble Phantom 6DOF 11 CSB Group Library for Input Devices in VR Environments Countless VR devices Different programing interfaces OS dependencies Application bound to expensive libraries Abstract Layer necessary Layer between application and VR devices Compact C library 12 CSB Group 6
Library for Input Devices in VR Environments Device driver concept Application OS PHANToM Game C. Fastrak Loads GHOST DirectInput 13 CSB Group Library for Input Devices in VR Environments Device driver interface Position Matrix Matrix Button Force Motortemp. Bool Float Float Bool Float Float Application Modularity OS independent Replace device without recompilation of application Simplified development process 14 CSB Group 7
Library for Input Devices in VR Environments Network Application Net. Server Fastrak Net. Client Network PHANToM GHOST Computer A Computer B 15 CSB Group SenSitus Haptic rendering for interactive fitting of xtal structures into EM density maps Idea: Correlation coefficient used for force calculation Guide user to better fitting location Additional source of information beside visual information User can overcome force User = global maximum, force = local maximum Problem: Too slow! Simplification of structural information 16 CSB Group 8
SenSitus Vector Quantization Vector Quantization Array of codebook vectors, describing a signal Signal-values replaced by nearest CV Voronoi tessellation 17 CSB Group SenSitus Vector Quantization Error: Sum of distance Problem: Find CV set with minimal error Solution: Topology Representing Networks Martinez, Schulten Neuronal network training process creates a Voronoi tessellation and Delaunay triangulation 18 CSB Group 9
SenSitus Reduced Fitting Criterion Correlation coefficient ρ (, r RT, ) calc ρem( r) 19 CSB Group SenSitus Reduced Fitting Criterion Reduced Model: Simplified Correlation Coefficient: wi ( R, T) 20 CSB Group 10
SenSitus Force Calculation Molecule: Rigid body in conservative force field C negative potential energy: Force field: Force on CV: Force on COM: Torque about COM: 21 CSB Group SenSitus - Speed Speed of force calculation Force calculation thread runs parallel to visual rendering Achievable force update rates 1kHz 10khz 1CPU 2CPU 3000 10000 2500 10 CV 9000 8000 10 CV Force-updates per second 2000 1500 1000 500 20 CV 40 CV Force-updates per second 7000 6000 5000 4000 3000 2000 20 CV 40 CV 1000 0 0 10 20 30 40 50 60 70 Time [s] 0 0 10 20 30 40 50 60 70 Time [s] 22 CSB Group 11
SenSitus Codebook Vectors Precision of the force calculation as a function of model complexity Actin Deoxyribonuclease I Complex (1ATN) - 5020 atoms 10CV 20CV 40CV 100CV 23 CSB Group SenSitus Translational Accuracy 100 90 80 70 Force 60 50 40 10CV 20CV 40CV 100CV 150CV 200CV 30 20 10 0-10 -9-8 -7-6 -5-4 -3-2 -1 0 1 2 3 4 5 6 7 8 9 10 Deviation in Ångström 24 CSB Group 12
SenSitus Overview SenSitus SVT Qt OpenGL Phantom Fastrak Net Portable: Versions for SGI, SUN, DEC, Linux, Windows 25 CSB Group SenSitus 26 CSB Group 13
SenSitus Molecular Structures PDB (Protein Data Bank) files Bond information often missing Guessing bonds by atom-atom distance criterion 2 Problem: Calculation complexity O(N )! Space partitioning approach є 27 CSB Group SenSitus Molecular Structures PDB (Protein Data Bank) files 28 CSB Group 14
SenSitus Molecular Structures Combination of drawing modes 29 CSB Group SenSitus Structural Data PSF (Protein Structure) files Canbeusedtogether with PDB files Used in Situs to store the codebook-vector connections 30 CSB Group 15
SenSitus Molecular Dynamics Visualizing computer simulations of molecular motion (DCD file format) 31 CSB Group SenSitus Volumetric Data 32 CSB Group 16
SenSitus - Future Enhancements Flexible fitting Interactive flexible fitting Haptic rendering of the biophysical restrictions during flexible fitting Integration of pure algorithmic solutions Online supervision Editing of search space Comparison of interactive and algorithmic results Interactive manipulation of datasets 33 CSB Group SenSitus Try it out yourself! Group I: Tuesday 2:45PM Group II: Wednesday 2:45PM 34 CSB Group 17
Acknowledgements Research Centre Jülich John von Neumann Institute for Computing Forschungszentrum Jülich The Scripps Research Institute Computational Structural Biology Group 35 CSB Group 18