2011.08.16 Medical VR Current Status and Future of Medical Virtual Reality Naoto KUME, Ph.D. Assistant Professor of Kyoto University Hospital 1. History of Medical Virtual Reality Virtual reality (VR) arrived early 70s. Myron Krueger demonstrated an interactive art for a proof of concept. After 40 years, flight simulators are inevitable for pilot training nowadays. Also 3D games with rich visual effects are common in these days. From a view point of medical training, VR of training use seems to have a huge potential for patient safety. Last 30 years, doctors and medical engineers struggled with development of medical VR simulators. However, it does not effectively work for training up to now, especially in Japan. Probably one reason is the required reality that doctors want to have an experience almost the same feeling in operation. It is an obvious defeat for VR against the real. Until researchers in medical VR domain recognize the simple theory that artificial reality never achieve the real, primary topic of the development of simulators was how realistic simulation can be provided. Our research was also dedicated for realistic simulation which should be given by physics based simulation. Eventually, the topic of realistic simulation was postponed in the research domain of medical VR. Primary research focus shifted to a discussion what you should essentially teach to medical students on course training. In such a circumstance, drastic change was occurred by European Surgical Institute (ESI) and other facilities in Europe around 2005 [1]. They started to use simulators on medical course. Using VR simulator is now mandatory to proceed with training. VR simulators are now going to be replaced with Box trainers gradually. Another tailwind for medical VR comes from robotics. Surgical robots such as Da Vinci and Zeus are popularizing [2]. Robot console which equips a video and manipulation interface is very suitable to operate both virtual environment and real environment by the same way. Therefore, medical VR is extended the possibility to support minimally invasive surgery by robot surgery. In the near future, simulation before robot surgery could be must for deliberation of surgery planning.
2. Overview Our research technique stands on physics based simulation. Early development is dedicated to present a realistic phenomenon of virtual organs with visual and haptic feedback. Skills in surgery are parted as surgical procedures: palpation, incision, ablation, puncture, suturing, etc. A big assumption is that if you well done of all procedures, you may achieve good operation. However, the result denied the assumption. You have to learn motor-skills which are required to know how to move your body as well as medical knowledge. Therefore, skill transfer is one of our research focus how to retrieve skills in operation. Teaching method on simulation and operation scenario editor is a key technology to build up simulation based training course. Another topic is a collaboration of medical VR and natural language processing (NLP) [3]. Doctors learn operations by operation manual books which must includes all information needed to achieve operation. Basic setup of equipments, procedures and annotations are given by the manuals. And patient specific information could be given by hospital information system (HIS). Therefore, we developed a system that NLP analyses manuals and retrieve parameters of what is needed in the operation. Then the parameters are given to VR simulator. Eventually, an initial setup of VR simulation is automatically completed. We defined the parameter format based on Medical Markup Language (MML). Even though the MML extension is not officially approved, we suggest that a kind of standard exchange format is required for simulation based education as well as electric health record (EHR). Finally, our further goal is to provide a communication tool of surgical skills by VR simulation environment in future. Even though the topic motives, the goal is far because the pervasive speed of robot surgery could be bottle neck for practical pervasiveness of medical VR simulators. Therefore, current primary topic is practical use of our technology in clinics to prove availability of medical VR. Especially, brain shift simulation, introduced in the following section, is well desired in neurosurgery navigation. The preliminary result was just release last April. Practical use of medical VR would be our target in the next decade.
3. Research Topics This section introduces past research topics in Yoshihara Laboratory in the last decade. The most of topics provide the simulation of surgical procedures. Also, some augmented reality based researches are given. Distributed massive simulation is a secondary topic to achieve real-time simulation by massive dataset. Explanations of following figures are given in the comment area of each figure. Figure 1: General medical VR simulator. The simulator consists with haptic feedback device, visual feedback, and simulation computer. In this figure, user manipulates two haptic devices. The photo is taken at MMVR 13(2005).
Figure 2: Palpation simulation [4]. Red spheres are the manipulation point such as fingertips. Arch of aorta is palpated before open heart surgery for clipping aorta. Stiffness of aorta is examined by pinching. The simulator provides force feedback of the stiffness: normal aorta, abnormal aorta. Finite element method (FEM) is employed for real-time deformation. 2003 Figure 3: Rectum palpation [5]. Two of soft tissues are contacted. The red sphere is a fingertip. User palpates the prostate shown at the left via the rectum shown at the right. 2006.
Figure 4: Haptic navigation on neurosurgery [6]. Crucial nerves lay in front of tumor which is illustrated as a sphere. When your manipulation point is too close to the nerves, a repulsion force is generated and prevents to touch nerve. 2004. Figure 5: Incision simulation [7]. Skin incision is simulated with smooth cutting edge. Although haptic feedback is not generated, incision procedure is achieved interactive manipulation. 2003. Figure 6: Ablation simulation [8]. Tearing soft tissue is physically simulated in real-time. Force feedback is given along to manipulation. 2005.
Figure 7: Architecture design of distributed massive simulation [9]. Computational complexity of soft tissue destruction requires huge amount of simulation resources. Parallel computing and the other acceleration method are needed for massive simulation, especially for the simulation of ablation. 2007. Figure 8: Transparent real-time volume rendering [10]. Acceleration method of visualization achieves real-time volume rendering and deformation on general laptop PC. 2006.
Figure 9: Open heart surgery planning [11]. Procedures are sequenced as cutting, opening and touch. Each step of the procedure consists of incision, deformation and palpation. You may change the approach of open heart anyhow. In this figure, median incision pattern and sternal splitting incision pattern are shown. 2006 Figure 10: Annotation on palpation. User mimics expert s activity on the simulated environment. Short memory of the procedure is not well improved. However, long term memory of the procedure is significantly improved in a case. 2007.
Figure 11: Annotation framework on virtual environment [12]. Annotation of reificative features is needed which can not be observed from externally. Even if your finger is stopped on a table, there are some reificative features such as how hard you push the table, which direction you try to move your finger. 2005. Figure 12: Simulation record editor [13]. Passed trajectory of expert s manipulation is given by blue line. Future trajectory is given by purple line. Training course provider picks up meaningful scene from the recorded sequence. Key technology is not trajectory tracking but dynamic annotation on deformable object. Expert s reaction on deformable object can be recorded. 2009.
(a) (b) Figure 13: Brain-shift simulation. (a) CT combined MRI image. Before deformation. (b) After deformation. On neurosurgery, brain subsides in the skull. The subsidence is called brain-shift. The cause is not well analyzed so far. We simulate the deformation by physics based gravity simulation. 2011. Reference information 1) Europian Surgical Institute, http://www.esi-online.de/, (Last access 2011.08.16). 2) Intuitive Surgical: da Vinci, http://www.intuitivesurgical.com/, (Last access 2011.08.16). 3) T. Takemura, Y. Kuroda, N. Kume, K. Okamoto, K. Hori, M. Nakao, T. Kuroda, H. Yoshihara: Requirement Extraction from Surgical Textbook using Natural Language Processing for Educational Virtual Reality Simulator. Japan Journal of Medical Informatics, vol.25, no.6, pp.457-462 (2006/06/20) Japanese. 4) Y. Yamamoto, M. Nakao, T. Kuroda, Hiroshi Oyama, M. Komori, T. Matsuda, G. Sakaguchi, M. Komeda, T. Takahashi: Palpation Simulator of Beating Aorta for Cardiovascular Surgery Training. Transactions on the Institute of Electrical Engineers of Japan E, vol.123, no.3, pp.85-92 (2003/03/01) Japanese. 5) Y. Kuroda, M. Nakao, T. Kuroda, Hiroshi Oyama, H. Yoshihara: Study of Spatial Anisotropy in Finger's Haptic Perception for Advanced Palpation Training. International Journal of Computer Assisted Radiology and Surgery, p.495 (2006/06/28) Osaka/Japan. 6) M. Nakao, K. Imanishi, T. Kuroda, H. Oyama: Practical Haptic Navigation with Clickable 3D
Region Input Interface for Supporting Master-Slave Type Robotic Surgery. Studies in health technology and informatics, vol.98, pp.265-271, IOS Press (2004/01/17) Newport Beach/USA. 7) M. Nakao, T. Kuroda, H. Oyama, Masaru Komori, Tetsuya Matsuda, T. Takahashi: Physically Based Fine and Interactive Soft Tissue Cutting. IPSJ Journal, vol.44, no.8, pp.2255-2265 (2003/08/01) Japanese. 8) N. Kume, M. Nakao, T. Kuroda, H. Yoshihara, M. Komori: Simulation of Soft Tissue Ablation for a VR Simulator. Transactions of the Japanese Society for Medical and Biological Engineering, vol.43, no.1, pp.76-84 (2005/04/01) Japanese. 9) N. Kume, Y. Kuroda, M. Nakao, T. Kuroda, H. Yoshihara, S. Mori, S. Tomita: Estimation of Application Level Speculative Operation on Distributed System for Haptic VR Simulatio of Invasive Operation. International Journal of Computer Assisted Radiology and Surgery, vol.2, no.1, S179-S180 (2007/06/28) Berlin/Germany. 10) M. Nakao, T. Matsuyuki, T. Kuroda, Kotaro Minato: Physics-based Manipulation of Volumetric Images for Preoperative Surgical Simulation. Proceedings of Asian Simulation Conference, pp.377-380 (2006/10/30) Tokyo/Japan. 11) M. Nakao, T. Kuroda, H. Oyama, G. Sakaguchi, M. Komeda: Physics-Based Simulation of Surgical Fields for Preoperative Strategic Planning. Journal of Medical Systems, vol.30, no.5, pp.371-380 (2006/10/01) 12) M. Rissanen, N. Kume, Y. Kuroda, M. Nakao, T. Kuroda, H. Yoshihara: Framework for Annotation of Haptic Data in Simulated Surgical Procedures. Proceedings of International Conference on Virtual Systems and Multimedia, pp.647-656 (2005/10/06) Ghent/Belgium. 13) N. Kume, M. Rissanen, Y. Kuroda, M. Nakao, T. Takemura, H. Yoshihara, T. Kuroda, S. Mori, S. Tomita, K. Yoshimura: Evaluation of Teaching Digital Rectal Examination on Annotated Haptic VR Simulator. Medical Virtual Reality, vol.7, no.1, pp.24-36 (2009/03/01) Japanese.