Interactive stereoscopic virtual reality: a new tool for neurosurgical education

Transcription

1 Use your 3D glasses from the December 2001 issue to view images in this article. J Neurosurg 96: , 2002 Interactive stereoscopic virtual reality: a new tool for neurosurgical education Technical note JEFFREY S. HENN, M.D., G. MICHAEL LEMOLE, JR., M.D., MAURO A. T. FERREIRA, M.D., L. FERNANDO GONZALEZ, M.D., MARK SCHORNAK, M.S., MARK C. PREUL, M.D., AND ROBERT F. SPETZLER, M.D. Division of Neurological Surgery, Barrow Neurological Institute, St. Joseph s Hospital and Medical Center, Phoenix, Arizona The goal of this study was to develop a new method for neurosurgical education based on interactive stereoscopic virtual reality (ISVR). Interactive stereoscopic virtual reality can be used to recreate the three-dimensional (3D) experience of neurosurgical approaches much more realistically than standard educational methods. The demonstration of complex 3D relationships is unrivaled and easily combined with interactive learning and multimedia capabilities. Interactive stereoscopic virtual reality permits the accurate recreation of neurosurgical approaches through integration of several forms of stereoscopic multimedia (video, interactive anatomy, and computer-rendered animations). The content explored using ISVR is obtained through a combination of approach-based cadaver dissections, live surgical images and videos, and computer-rendered animations. These media are combined through an interactive software interface to demonstrate key aspects of a neurosurgical approach (for example, patient positioning, draping, incision, individual surgical steps, alternative steps, relevant anatomy). The ISVR platform is designed for use on a desktop personal computer with newly developed, inexpensive, platform-independent shutter glasses. Interactive stereoscopic virtual reality has been used to capture the anatomy and methods of several neurosurgical approaches. In this paper the authors report their experience with ISVR and describe its potential advantages. The success of a neurosurgical approach is contingent on the mastery of complex, 3D anatomy. A new technology for neurosurgical education, ISVR can improve understanding and speed the learning process. It is an effective tool for neurosurgical education, bridging the substantial gap between textbooks and intraoperative training. KEY WORDS neurosurgical education neurosurgical simulation stereoscopy virtual reality QuickTime virtual reality computer-aided learning surgical anatomy T Abbreviations used in this paper: ISVR = interactive stereoscopic virtual reality; MR = magnetic resonance; OR = operating room; QTVR = QuickTime Virtual Reality; 2D = two-dimensional; 3D = three-dimensional. 144 HE process of learning neurosurgery is difficult due to both the inherent complexity of the subject and the limitations of standard educational methods. The ultimate success of a neurosurgical approach is contingent on a mastery of complex, 3D anatomy. Although the intricacy of the anatomy is a given, new computer-based educational methods can improve understanding of this anatomy and hasten the process of learning. The time-proven standard for neurosurgical education is a combination of textbooks, cadaver dissection, and intraoperative training. These tools are effective, but each has intrinsic disadvantages. Textbook-based anatomy is 2D, limited to fixed views, and difficult to extrapolate to views encountered during a surgical approach. Even surgical atlases based on 2D images obtained from a surgical perspective fall short of representing intricate 3D structures. Furthermore, although multiple images are often used to represent anatomical relationships, the spatial correlation between these images is not obvious. Consequently, although textbooks provide an important foundation for neurosurgical education, the need to augment learning with other tools is significant. Cadaver dissection is an invaluable aid to learn neurosurgical anatomy and techniques. The process is interactive, 3D, and readily applied to the OR setting. Unfortunately, several practical limitations exist: the availability of cadavers is limited, costs (such as those for preparation, facilities, instructors, and instruments) are considerable, and students must commit a substantial amount of time. As a result, cadaver dissection typically accounts for only a small fraction of a neurosurgical resident s education. Neurosurgical anatomy and techniques are ultimately learned in the OR through an apprentice-type relationship with a senior surgeon. The anatomy and skills learned in this setting form the foundation for a neurosurgeon s career.

2 Interactive stereoscopic virtual reality This type of learning is the gold standard for neurosurgical education, but it too has relative disadvantages. Learning in the OR tends to be relatively high pressured and time limited. In addition, the anatomy can be exposed only as clinically warranted. The opportunity to study 3D anatomy and surgical techniques outside the OR would eliminate many of these disadvantages. 5,6 Intraoperative training would be facilitated, and students would have the opportunity to learn at their own paces. 5 Furthermore, such a venue would optimize the educational experience in ways not possible in an OR setting. 5 Several elegant surgical simulation systems have already been developed. 1,2,4,7,8 Most rely on a combination of imaging and computer-based 3D rendering. Although useful, the costs of these systems can be prohibitive, their availability restricted, and the visual experience they provide limited. The processing power required for real-time or near-real-time 3D rendering is substantial. Systems that incorporate stereoscopic visualization require special displays or projectors; as a result, surgical simulation systems are expensive and typically require students to have access to specialized computers and equipment. 3 Even the best 3D-rendered images cannot compare with the visual experience of seeing anatomy in the OR. 7 We have developed a new tool for neurosurgical education known as ISVR. This imaging platform allows unrivaled demonstration of complex 3D anatomy and relationships. Interactive stereoscopic virtual reality can be used to recreate the experience of neurosurgical approaches much more realistically than standard educational methods. The platform is designed to be inexpensive, widely distributable, easily implemented, and readily scalable. Technical Components of ISVR Interactive stereoscopic virtual reality permits the accurate recreation of neurosurgical approaches through integration of several forms of stereoscopic multimedia (video, interactive anatomy, and computer-rendered animations). The content explored using ISVR is obtained through a combination of approach-based cadaver dissections, live surgical images and videos, and computer-rendered animations. These media are combined through an interactive software interface to demonstrate key aspects of a neurosurgical approach (for example, patient positioning, draping, incision, individual surgical steps, alternative steps, relevant anatomy). Interactive stereoscopic virtual reality is designed for use on a desktop personal computer with newly developed, inexpensive, platform-independent shutter glasses. Stereoscopic Video Stereoscopic video, a key element of the ISVR platform, is acquired during live surgery and approach-based cadaver dissections. Simulated surgical approaches performed during cadaver dissection demonstrate techniques and anatomical relationships from a surgical perspective. For example, cranial neurosurgical approaches can be reproduced accurately with regard to the patient s head position, scalp incision, craniotomy, intracranial dissection, and anatomy. Cadaver specimens are procured and prepared using standard procedures, including intravascular injection of dyed silicone for ease in distinguishing venous and arterial structures and to create a more lifelike dissection. Stereoscopic video is captured using an operating microscope and a stereoscopic camera, and it is recorded using a digital video recorder. After the video is transferred to a computer workstation, it is edited and processed for stereoscopic computer display. The footage is organized into several short clips, each used to demonstrate a particular step in the neurosurgical approach. These clips are incorporated into the ISVR platform so that viewers can study any specific aspect of a neurosurgical approach or review the clips sequentially. The software interface provides helpful controls for each video clip, including pause, play, rewind, and fast forward. Interactive Stereoscopic Anatomy Interactive stereoscopic anatomy is a new tool. Multiple photographic digital images of anatomy are obtained, with each image representing the view from a slightly different angle. Combining these images by using commercially available software creates a form of virtual reality (Fig. 1). We use a popular platform, QTVR. Sequential display of the appropriate images allows the user to control the view by panning (moving right and left) or tilting (moving up and down). Unlike demanding 3D computer renderings, QTVR requires only standard digital images that can be viewed on a typical desktop computer. Furthermore, the visual experience is considerably more lifelike than computer-rendered images. Three-dimensional information is gained through passive visual cues (such as lighting, shadow, and angle of sweep), and the perception of depth is provided by multiple object views. The addition of actual stereoscopic images to the QTVR platform creates an even more dramatic effect. This combination of true stereoscopic visualization with interactive capability creates a powerful form of personal computer based virtual reality. The user sees 3D anatomy and is able to control its orientation. The stereoscopic images are obtained using digital cameras, which are mounted to a robotically controlled surgical microscope. This microscope allows anatomy to be viewed from carefully defined, reproducible trajectories. After the microscope is positioned, two digital images (one from each eyepiece) are captured and transferred to a computer workstation. The microscope is moved to a new position, and the next set of images is captured. This process is repeated until the entire sequence of images is obtained. The images obtained through both right and left eyepieces at each position are combined into a composite stereoscopic image (Fig. 2). The composite images from each position are then used to create an interactive sequence. Several variables can be controlled when capturing images in this manner: angle between images, angle of maximum tilt and pan, number of rows and columns, magnification, focal length, radius of microscope, spherical rotation, and image resolution. By manipulating these variables, the final QTVR movie and the viewer s experience can be tailored. For example, one QTVR movie can be created to allow the user to scan the entire exposure of an approach while another movie can allow the user to rotate around key anatomical structures. All images are captured 145

3 J. S. Henn, et al. Multimedia Interface The software interface, a critical part of the ISVR experience, provides interactive control over the multimedia content. The result is an active learning experience that can be individually tailored. The ISVR interface was created using authoring software. The interaction is controlled with a menu-driven interface through which the multimedia elements are combined into a unified interactive experience. Beyond providing control of the content, the ISVR interface incorporates numerous other multimedia capabilities: synchronized voice narration, interactive anatomy identification, hyperlinks to references, and built-in self-assessment modules. FIG. 1. Conceptual demonstration of interactive virtual reality. A sample 3D object is represented by nine 2D images, each obtained from a slightly different perspective. These images can be combined to create a form of virtual reality. through the microscope, and an adapter lens that enlarges the field of view can be used for larger regions of anatomy. Currently, stereoscopic interactive anatomy sequences are obtained using cadaver dissections. We are developing procedures to capture these sequences during live surgery. Computer-Rendered Stereoscopic Animations Computer-rendered stereoscopic animations are another integral component of the ISVR platform. Animation sequences are ideal for demonstrating aspects of a neurosurgical approach that are not well illustrated using traditional methods. Initially, these animations are generated as 3Drendered objects on a high-performance graphics workstation. The 3D-rendered objects are then used to create stereoscopic animations (such as image, video, and QTVR sequence). For example, animations have been created to show head positioning during various cranial approaches. During the design step, the scalp is made semitransparent to demonstrate the position of the underlying skull during positioning. In this manner, animations can be used to highlight anatomical relationships and surgical techniques in ways not possible through cadaver dissection or live surgery. Computer-Based Stereoscopic Display Interactive stereoscopic virtual reality is designed for use on a standard desktop personal computer (Fig. 3). The platform is inexpensive, widely distributable, and easily implemented. It is based on multimedia content that can only be experienced using a computer interface. For example, stereoscopic interactive anatomy and computer animations are inherently computer based. Although stereoscopic video can be played using dedicated video equipment, computerbased video offers several advantages including ease of implementation, interactive capability, and a menu-driven interface. The ability to visualize anatomy and surgical techniques stereoscopically is invaluable. The intricate 3D relationships of complex neurovascular anatomy are difficult to portray with 2D images. Furthermore, the immersive experience provided by stereoscopic visualization more accurately recreates the experience of surgery than other alternatives. Stereoscopic vision relies on both monocular cues and binocular disparity. Monocular cues, which can be helpful at all distances, include object familiarity, interposition, linear and size perspectives, illumination, and motion parallax. Binocular disparity, which predominates for close objects, allows the brain to compare slightly different views of the same object. The result is interpreted as depth perception. The content of ISVR (images, video, interactive anatomy, and computer animations) is displayed using stereoscopic (binocular) images, resulting in a 3D experience. In a sense, however, the final ISVR experience is four dimensional. Stereoscopic video and animations show anatomy in three dimensions, with the added fourth dimension of time. Stereoscopic interactive anatomy sequences likewise provide a fourth dimension, the interactive trajectory selection (thus capitalizing on both monocular and binocular cues). The stereoscopic display, a defining part of ISVR, is implemented through the use of liquid crystal display shutter glasses, although other systems could be used as well (Fig. 3). The stereoscopic effect is achieved using binocular images, allowing the left eye to see only the left-eye image and the right eye to see only the right-eye image. We use inexpensive ( $40) platform-independent shutter glasses for computer stereoscopic display. They easily connect between computer and monitor and require no software, special graphics cards, or internal alterations to the computer. When activated, the glasses alternatively darken the liquid crystal display over each eye while synchronously blank- 146

4 Interactive stereoscopic virtual reality FIG. 2. Demonstration of interactive stereoscopic anatomy viewed during a far-lateral approach. Upper: An artist s conception of the approach showing two photographic trajectories. Lower: Stereoscopic images were obtained from each trajectory. The stereoscopic effect can be seen by wearing red-cyan anaglyphic glasses. Shutter glasses are used for the computer-based version to improve color resolution. Multiple stereoscopic images are combined into an interactive interface to create a virtual reality experience. 147

5 J. S. Henn, et al. FIG. 3. Photograph providing a conceptual demonstration of the ISVR experience. The platform is based on the use of a desktop personal computer and inexpensive shutter glasses. The effect for the viewer is to see and interact with stereoscopic approach-based neurosurgical anatomy. ing alternating horizontal lines on the computer monitor (the shuttering rate is determined by the refresh rate of the computer display). Stereoscopic content is formatted using a horizontal interlaced format. Two images (one for each eye) are combined to create a single composite image, such that alternating horizontal lines originate from each of the two images. When the shutter glasses are active, the left eye sees only the even lines (made from the left-eye image) and the right eye sees only the odd lines (made from the right-eye image). The result is stereoscopic visualization (Fig. 2). Color anaglyphic glasses also can be used for stereoscopic display, but the shutter glasses eliminate eye strain and make color appear normal. Sources of Supplies and Equipment Carl Zeiss, Inc. (Thornwood, NY) manufactured both the OPMI operating microscope and the MediLive stereoscopic camera. Sony Corp. (Tokyo, Japan) manufactured the digital video recorder. QuickTime Virtual Reality was developed by Apple Computers (Cupertino, CA). The stereoscopic images were obtained using digital cameras purchased from Pixera Corp. (Los Gatos, CA) and a Surgi- Scope robotically controlled surgical microscope provided by Surgical Navigation Technologies (Louisville, CO). Director authoring software, obtained from Macromedia, Inc. (San Francisco, CA), was used to create the ISVR interface. The shutter glasses (VR Visualizer) were acquired from Vrex, Inc. (Elmsford, NY). Applications of ISVR Current Applications We are using the ISVR platform to create a Virtual Atlas of Neurosurgical Approaches. This compact disk/digital video disk/internet based atlas will cover standard neurosurgical approaches and create an active virtual learning experience of anatomy and surgical techniques. The content of the platform is obtained in the manner described in Technical Components of ISVR and this is supplemented with clinical case examples, references, and other information. In addition to demonstrating approaches, another important goal of ISVR is to demonstrate specific pathological conditions and structures and their effects on normal anatomy. This goal is easily accomplished through stereoscopic video from live surgery and through stereoscopic computer animation. To include pathological conditions and structures in interactive anatomy sequences is more complex and requires the application of morphing procedures. Nonetheless, the result will be an even more valuable educational tool. The interface for each approach will follow a common theme and contain stereotypical navigation menus. Related approaches will be grouped into modules for both organization and comparison of procedures. For example, the module covering the pterional craniotomy will also include an orbitozygomatic osteotomy. In this manner, the interface will allow an interactive comparison of the exposure obtained using each approach. Interactive stereoscopic virtual reality can be applied equally well to cranial, spinal, and peripheral nerve procedures. In all cases, the demonstration of 3D anatomical relationships and surgical techniques is very useful. Procedures that involve the placement of instrumentation are well demonstrated using this same combination of stereoscopic video, interactive anatomy, and animation. In this way, ISVR can be used to study procedures as well as approaches. The application of ISVR to neuroendoscopy is also a natural direction to pursue. The optical physics of endoscopy allow accurate anatomical recreations using a relatively small number of images, and the computer-based interface of ISVR provides an accurate recreation of the endoscopic experience. Future Neurosurgical Applications Combining ISVR with frameless image-guided navigation techniques offers several important benefits. First, ISVR can be a useful educational tool when used to improve understanding of neurosurgical navigation. The image-based feedback improves understanding of the relationship between anatomical structures and their imaging counterparts. Another potential benefit of this combination would be an interface based on the correlation between MR imaging data and interactive stereoscopic anatomy. Using this method, the viewer could select a specific region on an MR image and be presented with a list of potential cranial approaches. It would then be possible to view the corresponding interactive stereoscopic anatomy for each approach, centered on the selected region of anatomy. In this manner, the viewer would gain a better understanding of surgical approaches with regard to relative exposure of anatomical regions of interest. This understanding would be reinforced with stereoscopic views from the appropriate trajectory and relevant surrounding anatomy. Another application of ISVR would be the creation of interactive stereoscopic anatomy sequences throughout every stage of a neurosurgical approach. For example, during a cranial approach, interactive stereoscopic anatomy se- 148

6 Interactive stereoscopic virtual reality quences could be obtained at the level of the scalp, skull, brain surface, and targeted anatomy. As long as the trajectories for image acquisition were maintained throughout the entire approach, it would be possible to create a multilayered, interactive stereoscopic anatomy sequence. The viewer would be able to rotate right and left or forward and backward within a given level of approach, and would also be able to move in or out of the various anatomical levels of the approach. Such a sequence would create a unique educational experience by demonstrating the anatomical relationships encountered when following a particular trajectory. Although ISVR is not based on true 3D renderings, a very lifelike experience could be created by combining the visual experience of ISVR with true 3D spatial calculations. This combination is particularly well suited to endoscopy. In the case of ventriculoscopy, for example, MR imaging data could be used to create a true 3D-rendered volume of intraventricular space. This 3D-rendered volume could be used to provide haptic (tactile) feedback. An actual endoscope could be connected to a haptic feedback device system, with movement of the endoscope resulting in appropriate ISVR displays. The visual experience would be based on the advantages of ISVR, whereas 3D modeling would provide the advantages of surgical simulation. Future Nonneurosurgical Applications Interactive stereoscopic virtual reality is a general platform that was developed for optimum demonstration of 3D anatomy and surgical techniques by using a standard personal computer. Although the platform was originally designed for approach-based neurosurgical education, it offers potential for several other valuable applications. The platform is well suited to recreating surgical approaches through an operating microscope. The ISVR platform can easily be translated to other microsurgical specialties, including ophthalmology, otolaryngology, vascular surgery, and peripheral nerve surgery. The potential for use in endoscopy could likewise be translated to urology, obstetrics and gynecology, orthopedics, and general surgery. For each of these specialties, the advantages would be similar to those for neurosurgery. The application of ISVR to surgical specialties that do not primarily use the operating microscope or endoscopy also has potential. One option would be to use an adaptor lens to allow larger anatomical structures to be well visualized through the operating microscope for the purpose of creating an ISVR experience. For even larger anatomical structures, a stereoscopic digital camcorder could be mounted to a robotic control system. In either case, the ISVR platform would fundamentally remain the same and would provide essentially the same advantages as those described for neurosurgery. The ISVR platform also is well suited to the display of anatomical relationships independent of surgical approach. For example, it would be useful to create interactive stereoscopic anatomy sequences for the study of any anatomical structure. Interactive stereoscopic virtual reality could be used to improve anatomy education for medical students, nursing students, therapists, anatomists, and medical illustrators. Finally, ISVR may have important uses outside medical education. The ISVR platform in general and the specific methods of content creation make it possible to create the same interactive stereoscopic experience for any small or microscopic object. This ability could have uses in advertising, marketing, cataloging, and high-technology industry. Regardless of the content being displayed, ISVR retains its advantages of being inexpensive, widely distributable, and easily implemented. Disclosure An application for a patent covering the ISVR technique has been submitted. References 1. Apuzzo MLJ: The Richard C. Schneider Lecture. New dimensions of neurosurgery in the realm of high technology: possibilities, practicalities, realities. Neurosurgery 38: , Bullitt E, Aylward S, Bernard EJ Jr, et al: Computer-assisted visualization of arteriovenous malformations on the home personal computer. Neurosurgery 48: , Dunkley P: Virtual reality in medical training (conference). Lancet 343:1218, Gorman PJ, Meier AH, Krummel TM: Simulation and virtual reality in surgical education: real or unreal? Arch Surg 134: , Haluck RS, Krummel TM: Computers and virtual reality for surgical education in the 21st century. Arch Surg 135: , Hoffman H, Vu D: Virtual reality: teaching tool of the twentyfirst century? Acad Med 72: , Kockro RA, Serra L, Tseng-Tsai Y, et al: Planning and simulation of neurosurgery in a virtual reality environment. Neurosurgery 46: , Weingaertner T, Hassfeld S, Dillman R: Virtual jaw: a 3D simulation for computer assisted surgery and education, in Westwood JD, Hoffman HM, Stredney D, et al (eds): Medicine Meets Virtual Reality. Amsterdam: IOS Press, 1998, pp Manuscript received June 19, Address reprint requests to: Robert F. Spetzler, M.D., c/o Neuroscience Publications, Barrow Neurological Institute, 350 West Thomas Road, Phoenix, Arizona neuropub@ chw.edu. 149