Lifesaving visualizations

Transcription

1 Lifesaving visualizations Interactive visualization in the design of Decision Support Systems Robert G. Belleman, PhD Computational Science University of Amsterdam R.G.Belleman@uva.nl

2 About us Scientific Visualization and Virtual Reality team part of Computational Science at UvA/IvI close collaboration with SARA Research theme: interactive visual exploration Software solutions and architectures, Problem Solving Environments, Interactive graphics devices Application areas: computational science (astro)physics, medicine, biology, finance, architecture, computer science,

3 Software solutions and architectures Software solutions and architectures Visualization as a Service: remote visualization on high performance computing resources High performance visualization on low performance devices Interactive Problem Solving Environments Visual exploration of large parameter/data spaces Integrated services for visualization and on-demand simulation Augmented Reality visualization of DTI fiber tracks on a TabletPC Interactive parameter exploration of the ACGT Oncosimulator in the OncoRecipesheet

4 Interactive graphics devices Microsoft Surface SARA Tiled Panel Display UvA DRIVE UvA PSS UvA WiVR UvA MTT

5 Overview Outline: The role of interactive visualization in Decision Support Systems Lessons learned Applications: patient specific disease treatment: simulated vascular reconstruction, cancer treatment prevention and/or containment of flood disasters social-genetic analysis of infectious diseases

6 Decision support Characteristics: Non-trivial problems Presentation of current state Information integration and presentation Human-in-the-loop Domain expert What-if? Challenges: Data deluge Heterogeneous, complex, dynamic Searching large parameter spaces Urgency

7 Simulated vascular reconstruction

8 Vascular disease Stenosis: Treatment: thrombolysis, balloon angioplasty, stent placement, endarterectomy, bypass Aneurysm: Treatment: shunt, bypass

9 The problem Best treatment often not obvious Human body is a complex structure A treatment is not always best under all situations Preop AFB w/ E-S Prox. Anast. AFB w/ E-E Prox. Anast. Angio w/ Fem-Fem Angio w/ Fem-Fem & Fem-Pop

10 Traditional treatment of vascular disease

11 Interactive simulated vascular surgery

12 Simulated Vascular Reconstruction Simulated vascular reconstruction Patient specific angiography data Fluid flow simulation software Simulation of reconstructive surgical procedure in VR Interactive visualization of simulation results in VR Pre-operative planning Explore multiple reconstruction procedures

13 Parallel fluid flow simulation Lattice Boltzmann Method (LBM) Lattice based particle method Regular lattice, similar to CT or MRI datasets Allows irregular 3D geometry Allows changes at run-time Velocity, pressure and shear stress calculated from particle densities Non-compressible homogeneous fluid, laminar flow Spatial and temporal locality Ideal for parallel implementation

14 Interactive exploration in VR CAVE, Personal Space Station (PSS) Visualize simulation results Flow field, pressure, shear stress, at run-time Interactive grid editing Simulate vascular reconstruction procedure Interactive exploration VR interaction to locate regions of interest

15 Interactive exploration in VR CAVE, Personal Space Station (PSS) Visualize simulation results Flow field, pressure, shear stress, at run-time Interactive grid editing Simulate vascular reconstruction procedure Interactive exploration VR interaction to locate regions of interest

16 Results Major achievements: Architecture for distributed interactive simulation and visualization Complicated framework using the High Level Architecture (HLA) Interactive visualization in VR environments Lessons learned: High visual impact Good PR... Domain experts are hard to please... Not considered intuitive, or even easy to use VR is too difficult to use CAVE is too immobile

17 ACGT: Advancing Clinico Genomic Trials on Cancer

18 In Silico Oncology Interactive 3D/4D visualization of In Silico Oncology simulations Collaboration ICCS/NTUA and UvA 3D volumes 96 x 96 x 96 voxels per timestep Tens to hundreds of timesteps Tumour layers : Proliferating cell layer Dormant cell layer Dead cell layer (necrotic core) Treatment depends on: Patient specific pathology Type of treatment (25+ parameters) Cell behaviour (50+ parameters)

19 Oncosimulator demonstration: component collaboration diagram user Recipesheet/ web portal 1 3 Simulation parameters Grid Resource Management System Job identifier (GRMS) 2 4 Job submission Job monitoring simulation parameters PSNC Oncosimulator Images 8 Visualization parameters job identifier vis parameters Visualization services Reference to: segmentation data by patient id 5 Data request 11 6 Data request s References to: DICOM images (before/after CT) by patient id segmentation data by patient id (intermediate) oncosimulator data by job id 9 Data 10 Segmentation data 7 (intermediate) oncosimulator data Data Management System (DMS)

20 Interactive parameter exploration Automatic job execution on grid architecture Interactive visualization as a service used by Web based applications (portal) Hokaido RecipeSheet: Parameter space exploration Personal Space Station: Reach in and touch your data Web based visualization Hokaido RecipeSheet Personal Space Station (PSS)

21 OncoRecipesheet Interactive parameter exploration of the ACGT Oncosimulator in the OncoRecipesheet (with Aran Lunzer, Hokaido Univ, Japan).

22

23 Results Major achievements: Parameter exploration on a Grid-based architecture Automatic job execution Web-based visualization services Visualization as a Service Serves multiple visualization front-ends from the same set of services Collaborative visualization High-end visualization on lowend devices Lessons learned: High visual impact Good PR... Domain experts are hard to please... Not considered intuitive, or even easy to use The gap is widening

24 Urbanflood Sensor systems, numerical models, computational simulation and scientific visualization for the development of Early Warning Systems

25 Context

26 Context Total length primary flood defenses in Netherlands: 2875km spread over 90 dike rings

27 Context Source: Assessment of primary flood defences in The Netherlands, Inspectie Verkeer en Waterstaat, National Report on 2006.

28 Worst Possible Flood West coast scenario (source: Rijkswaterstaat) 4 h 12 h 48 h 1 week Maximum waterdepth

29 UrbanFlood Online Early Warning System Public S S S S S S Control Centre Control Centre

30 UrbanFlood cascade Cross section of dike with sensors Seismic (acoustic) Water in/outlets Fibre optics Basin... Inverted pendulum Fibre optics Channel dike Macro stability dike Sensors + AI Structural analysis Breach modeling Population Dynamics Inundation simulation

31 Monitoring through sensors GeoBeads TM, Alert Solutions Pore pressure Temperature Inclination (2 DOF) Currently installed at 5 locations in NL St. Petersburg, RU Boston, UK AnySense

32 IJkdijk, Bellingwedde, Groningen

33

34

35

36

37 Livedijk Eemshaven Since Oct cross-sections, 230m apart, fibre optic cable

38 Sensor monitoring Reliability Analysis AI Anomaly Detection Virtual Dike High risk? Yes Breach simulator Structural analysis Decision Support System Risk Assessment Inundation simulator Population Dynamics

39 Structural analysis: the Virtual Dike FSI model: water flow through porous media + structural dynamics Partially saturated soils with water retention Finite element method Dike in Groningen, NL tidal water load and flood conditions sensor input

40 pressure head, mm Virtual Dike simulations Pressure dynamics Velocity vectors Virtual and real sensor dynamics 1E4 virtuale4(k=1e-9m2) virtuale4(k=1e-10m2) X : : :12

41 Discharge (m 3 /s) Breach prediction Hydrograph time

42 Inundation simulation Model: simplified shallow water equations Developed by HR Wallingford, UK DTM: AHN-1 Rijkswaterstaat (25m 2 /pixel) -7m 20m

43 Inundation simulation Amsterdam Science Park My office

44 Inundation simulation St. Petersburg, Russia Boston, UK

45 Decision Support System Integration of: Sensor data + AI Simulation results Maps, weather, ships, roadwork, traffic, location of emergency services,

46 Population Dynamics Life Safety Model (HR Wallingford) City evacuation

47 Multi-touch displays

48 Visualization as a Service Sensor data, simulation data, live streams Visualization server London Web access Visualization services data aggregation data transformation feature detection collaboration coordination St. Petersburg High performance visualization on low performance devices Amsterdam Simulations Groningen

49

50

51 Results Major achievements: Cloud-based simulation Information integration through Common Information Space Multitouch interfaces Web-based visualization services Lessons learned: Multitouch displays are easy to use, even by domain experts! The gap is closing again... Must remember to keep interaction simple

52 DynaNets

53 Research Question - Maximum likelihood analysis - Bayesian analysis Annotated HIV sequence data Sequence info. Demographic info. Social, demographical and clinical data analysis Phylogenetic tree Social tree From theoretical point of view, is it possible to come up with a mathematical formulation to infer phylogenetic tree from a social tree (network), and how redundant is that?

54 Building the Network: Reductionism approach Start with an undirected fully-connected network of HIV sequences (each node is connected to all other nodes). Then apply social and demographical filters to drop the socially impossible connections between every two nodes. Filtering Fully-connected Network Filtered Network

55 Filters Filter 1: Age difference Age difference for having sexual relationship (for MSMs and Heterosexuals) or social contact (for IDUs to share needle) Free parameter Filter 2: Mode of infection (m) & gender (g) For patients 1 and 2: If(g1 = g2 = Male && m1 = Homo, m2=hetero) connection = 0 If(g1 = g2 = Male && m1 = m2 = Homo) connection = 1 If(g1=Male, g2 = Female && m1 = m2 = Hetero) connection = 1. Filter 3: therapy date (t) & seroconversion date (s) Transmission probability decreases 80-98% after treatment [2][3]. For patients 1 and 2: If(t1 is older than s2) connection = 0 If(t2 is older than s1) connection = 0

56 Workflow HIV Sequence data - Annotated with social and genetic information - Unique patients Building Fully-connected network Fully-connected network - All patients are connected Filtering Filtered-network - Social filters (age difference, gender and mode of infection, seroconversion & therapy date) Seroconversion function Directed-filtered network - Direction from a patient with older sero date to a patient with more recent sero date (no loops) Patient with older seroconversion date Patient with newer seroconversion date

57 Twilight Interactive network visualization Overview of features: Cross platform Developed on Linux, Windows, known to work on Mac, Microsoft Surface Focus on interaction: the application invites exploratory analysis Navigation (zoom, pan) Selection (single node, rubber-band) Appearance (layout, size/colour/position of nodes, edges, labels) Supports visualization of large graphs and multiple graphs simultaneously Comparative visualization; let layout of one graph follow another Supports several layout algorithms random, circular, tree, Kamada-Kawai, Fruchterman- Reingold, Reingold-Tilford, graphopt Transitions between layouts are smooth to provide a visual cue of the chances Analysis methods Path finding, connectivity distribution, degree distribution, connected components Scripting interface to Python

58

59

60 Results Major achievements: Interactive graph visualization for large graphs Support for dynamic (time variant) graphs Interactive exploration through multitouch interface Lessons learned: Multitouch displays are easy to use by domain experts Thanks to ipad/iphone Move over effect Main challenge: mapping functionality to affordance

61 Conclusions Designing Decision Support Systems is challenging, but the potential is huge From distributed HPC, Grids to Clouds: nobody noticed Domain experts prefer familiar interfaces VR still has a long way to go

62 Robert G. Belleman, PhD Computational Science Universiteit van Amsterdam Science Park XH Amsterdam R.G.Belleman@uva.nl Web :