Communication Requirements of VR & Telemedicine Henry Fuchs UNC Chapel Hill 3 Nov 2016 NSF Workshop on Ultra-Low Latencies in Wireless Networks Support: NSF grants IIS-CHS-1423059 & HCC-CGV-1319567, CISCO, Microsoft, NVIDIA, and BeingThere Centre (ETH Zurich, NTU Singapore, UNC Chapel Hill, and SRF-IDMPO Singapore).
Types of Telemedicine Store and forward: no need for remote medical provider to be available when patient s data is transmitted (teledermatology, teleradiology) Remote monitoring: heart disease, diabetes, asthma Real-time interactive: Fixed camera and display: history review, psychiatric evaluations Mobile presence robots: visit patients in hospital rooms http://www.raagalahari.com/localevents/3045/rp-7- remote-presence-robot-launch-apollo-hospital-grouphyd.aspx Not much VR in traditional telemedicine (above) RP-7 Mobile Robot VR would provide a sense of immersive presence in the remote place https://en.wikipedia.org/wiki/telemedicine
Telemedicine for Surgery Remote surgery: Demo in 2001, surgeon in NY; patient in Strassbourg, France (cholecystectomy/gallbladder removal), with Zeus robotic system. (More in 2003.) Remote surgical consultation: head-worn video camera on local surgeon, second camera over operating table (US DoD, US and Iraq?) These aren t VR either; don t give the user an immersive sense of presence in a Virtual World (the remote place) stereo display for surgeon http://www.intersurgtech.com/resources/ images/group1/ overtheshoulderwide_03_pers.jpg http://robocatz.com/davinci-surgical.htm
VR for Telemedicine Providing an immersive sense of presence in the remote site Key capability for sense of presence: as many depth cues as possible: stereopsis, head-motion parallax, occlusion, focus cues. need a 3D data/model of the remote site, not just camera imagery Original imagery from multiple cameras 3D reconstruction from multiple camera imagery stereo walk-around VR visualization Cha,, Fuchs. Immersive Learning Experiences for Surgical Procedures, Medicine Meets VR 2016.
Most Advanced 3D Telepresence System: Microsoft Research Holoportation Orts-Escolano,..,Izadi (23 authors!) Holoportation: Virtual 3D Teleportation in Real-time, ACM UIST Symposium. Oct 16-19, 2016, Tokyo
Expansion of Telemedicine with VR Remote surgical consultation, patient history review, psychiatric evaluation,.. Communication & computation steps, for each frame (~ 1/60 sec): Multi-camera capture: 20-40 cameras (24 in Holoportation) Reconstruction of single frame of 3D scene/model from 20-40 images Transmit 3D model to remote site At remote site: render stereo image for user s current perspective Transmit the stereo image to headset Twice this load for system with two users, one at each site
Communication & Computation Needs Multi-camera capture: 20-40 cameras Reconstruction of 3D scene/model Transmit 3D model to remote site At remote site: render stereo image Transmit the stereo image to headset 20-40 video stream ~10 PC/GPUs 10-20 video equiv. 2 GPUs 2 video equiv. Twice this load for system with two users, one at each site
Most Important Link to Minimize Latency At remote site: render stereo image 2 GPUs Transmit the stereo image rendering to headset 2 Video frames If there is noticeable latency, when user s head moves, the images in the headset will be for a (slightly) older/different head position Gives illusion that the virtual scene is not fixed to the user s physical environment Causes nausea, cyber sickness
How Much Latency Is Tolerable? Latency from head-pose-capture to new-image-in-headset cenario: Conversation with local and remote individuals remote local Holoportation, UIST 2016 near-eye display local 150º / sec 2 M local remote Latency induced errors in perceived location of remote individuals total latency degrees moved arcmin nm. pixels 60-degree HD screen mm of offset error 10 millisec. 1.5 90 48 52.36 local remote 1 millisec. 0.15 9 4.8 5.24 100 microsec. 0.015 0.9 0.48 0.52
Overcoming Tracker-to-Display Lag Image seen thru AR HMD mechanical Head tracker Head pose 50 millisec. latency Application 11cm Rendering engine AR overlay Offset error Real checker board FPGAs (Xilinx) DLP micromirrors Post-rendering warp Display device Mechanical tracking rig & near-eye display ~ 100 microsec. latency Lincoln, Blate, Singh, Whitted, State, Lastra, Fuchs From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality IEEE VR 2016 Conf. (Best Paper Award) and IEEE TVCG
Overcoming Tracker-to-Display Lag (Video Demo) Head tracker Head pose Application Rendering engine (future) Post-rendering warp Display device Lincoln, Blate, Singh, Whitted, State, Lastra, Fuchs From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality IEEE VR 2016 Conf. (Best Paper Award) and IEEE TVCG
Predictable Latency Also Important Prediction is basic to strategies for ameliorate effects caused by latency Accurate prediction very challenging with varying latency
Additional Communication Needs Room-based cameras often insufficient for 3D scene capture Lots of occlusion in many realistic situations Additional ~50 cameras may be needed: in headset, in wristbands,.. Trauma bay at UNC Hospitals: fills up with 6-10 medical staff, the patient, equipment, often very close together Rohan Chabra
Conclusions: VR/Telemedicine will soon need Wide bandwidth, low and predictable latency communication paths 50 HD-4k video cameras steams Camera streams distributed among multiple CPU/GPU clusters Time streams of 3D scene models to distant collaborators 3D scene models to CPU/GPUs for rendering for headset displays Rendered images transmitted to headset. Image seen thru AR HMD THE END 50 millisec. latency AR overlay 11cm Offset error Real checker board nical tracking rig ~ 100 microsec. latency Holoportation, UIST 2016