Intraoperative stereoscopic QuickTime Virtual Reality

Size: px

Start display at page:

Download "Intraoperative stereoscopic QuickTime Virtual Reality"

Claribel George
5 years ago
Views:

1 See the Editorial and the Response in this issue, pp J Neurosurg 100: , 2004 Intraoperative stereoscopic QuickTime Virtual Reality ATTILA BALOGH, M.D., MARK C. PREUL, M.D., MARK SCHORNAK, M.S., MICHAEL HICKMAN, B.F.A., AND ROBERT F. SPETZLER, M.D. Division of Neurological Surgery and Neuroscience Publications, Barrow Neurological Institute, St. Joseph s Hospital and Medical Center, Phoenix, Arizona Object. The aim of this study was to acquire intraoperative images during neurosurgical procedures for later reconstruction into a stereoscopic image system (QuickTime Virtual Reality [QTVR]) that would improve visualization of complex neurosurgical procedures. Methods. A robotic microscope and digital cameras were used to acquire left and right image pairs during cranial surgery; a grid system facilitated image acquisition with the microscope. The surgeon determined a field of interest and a target or pivot point for image acquisition. Images were processed with commercially available software and hardware. Two-dimensional (2D) or interlaced left and right 2D images were reconstructed into a standard or stereoscopic QTVR format. Standard QTVR images were produced if stereoscopy was not needed. Intraoperative image sequences of regions of interest were captured in six patients. Relatively wide and deep dissections afford an opportunity for excellent QTVR production. Narrow or restricted surgical corridors can be reconstructed into the stereoscopic QTVR mode by using a keyhole mode of image acquisition. The stereoscopic effect is unimpressive with shallow or cortical surface dissections, which can be reconstructed into standard QTVR images. Conclusions. The QTVR system depicts multiple views of the same anatomy from different angles. By tilting, panning, or rotating the reconstructed images, the user can view a virtual three-dimensional tour of a neurosurgical dissection, with images acquired intraoperatively. The stereoscopic QTVR format provides depth to the montage. The system recreates the dissection environment almost completely and provides a superior anatomical frame of reference compared with the images captured by still or video photography in the operating room. KEY WORDS QuickTime Virtual Reality stereoscopy neurosurgical simulation surgical anatomy computer-aided learning neurosurgical education I NTRAOPERATIVE images captured through an operating microscope often leave viewers who were not present during surgery with an unsatisfactory experience. This situation involves several issues, as follows: 1) 2D images; 2) limited depth of field of the images, showing little of the surrounding anatomy; 3) a limited area in focus; and 4) user-passive images (that is, they are not incorporated into a software system that enables viewer-defined navigation). Consequently, the information is suboptimal for the purposes of education, review, and presentation. We have previously described a system that captures approach-based sequential images from exquisite laboratory cadaveric dissections and displays them using QTVR (Apple Computer, Inc., Cupertino, CA) in standard and stereoscopic mode. 9 As with standard QTVR, the images are captured and processed so that the user can later interact with and control the display of the surrounding environment (in this case anatomy) by panning (moving left to right) and tilting (moving up and down). The previously described stereoscopic image system improves visualization of complex neuroanatomy and permits Abbreviations used in this paper: CA = carotid artery; QTVR = QuickTime Virtual Reality; 2D = two-dimensional; 3D = three-dimensional. more complete review and use of laboratory cadaveric dissections. Consequently, we sought to expand the system to take advantage of useful time-sensitive image information available from the operating room. In this report we describe the intraoperative use of this system to capture and display images obtained in neurosurgical cases by using standard and stereoscopic QTVR modes. Technical Components After a disease structure had been identified or the region of interest had been dissected and hemostasis had been achieved during surgery, the operating microscope (model NC4; Carl Zeiss Surgical, Inc., Thornwood, NY) was replaced with a robotic microscope (Zeiss MKM carrier with OPMI neurosurgical microscope; Carl Zeiss Surgical, Inc.) equipped with two digital cameras (EOS D60; Canon, Inc., Tokyo, Japan) that were attached to the sidearms of the microscope to capture intraoperative images simultaneously. A grid system established to facilitate image acquisition with the Zeiss MKM microscope allowed convenient QTVR acquisition. Two points that corresponded to the two extreme angled positions in the operative field of interest were identified. Usually a midpoint between these extremes was chosen as the QTVR target point. The senior author (R.F.S.) determined the field of interest and set the target J. Neurosurg. / Volume 100 / April,

A. Balogh, et al. FIG. 1. Montage showing the method of inverted cone image acquisition. The tip of the cone is the pivot point, and the anatomical target is at the center of the grid.

2 A. Balogh, et al. FIG. 1. Montage showing the method of inverted cone image acquisition. The tip of the cone is the pivot point, and the anatomical target is at the center of the grid. point for acquiring the QTVR image. The size of the grid was changed by entering the number of the horizontal and vertical steps between the two locations. Each step of the grid represented the position to which the microscope would automatically move. Microscope pivot points were established using two methods. In one method a targeted structure or a low point in the dissection was used as the pivot point (for example, the bottom of an inverted cone; Fig. 1). This method was used for large, wide, and deep exposures. In the other (keyhole) method, the pivot point was above the target structure, but structures below the pivot point were still imaged (Fig. 2). This method was used in dissections with a narrow corridor. Regardless of the pivot points, images were acquired in focus by using the microscope s autofocus function. Two images were captured from footage shot by both cameras at each position for the left and right eyepieces. Simultaneous right and left images were acquired using two remote control devices. The vertical and horizontal extent of angulation from the focal point was approximately 45. The number of pictures captured in the grid was predefined by the number of points set into it indicating that an image would be obtained. Thus, the navigation experience became smoother as more images were obtained. Acquisition time, however, lengthened as the number of images acquired increased, because it took longer for the microscope to move through the grid. Each camera was equipped with a 10-GB microdrive (IBM Storage Systems Division, San Jose, CA) for image storage. Image Processing Images were processed to produce a standard or stereoscopic QTVR recording. Both types of image sets were usually produced. Producing Standard QTVR Images Images were processed with commercially available software and hardware. Images stored on the microdrives were downloaded to a graphics workstation by using Canon software drivers. Left and right images were archived in separate folders for later processing to a smaller size in Adobe Photoshop (Adobe Systems, Inc., San Jose, CA). The VR Worx program (VR Toolbox, Inc., Pittsburgh, PA) was used to reconstruct the image acquisition grid from the Zeiss MKM microscope on the graphics workstation. Images were loaded into the grid to match each point at which they were originally set; the Zeiss MKM QTVR acquisition software was used for this purpose. The VR Worx program was used to construct the QTVR image, which can be made using all of the left or right eyepiece images. Producing Stereoscopic QTVR Images Three-dimensional software (Stereo Image Factory Plus; SOFTreat Co., Raleigh, NC) was used to produce and align corresponding interlaced left and right image pairs. The interlaced pictures were archived in separate folders and loaded into the recreated grid with the aid of VR Worx. The final product was an interlaced QTVR image that could be viewed with synchronized liquid crystal display shutter glasses (VR Visualizer; Vrex, Inc., Elmsford, NY) or with a projector system equipped with polarized lenses to produce the stereoscopic effect. The shutter glasses used to connect the computer workstation and monitor require no specialized additions to the workstation such as graphic cards or software. Illustrative Cases Three cases illustrate the benefits and limitations of the standard and stereoscopic QTVR systems. 592 J. Neurosurg. / Volume 100 / April, 2004

3 Intraoperative stereoscopic QuickTime Virtual Reality FIG. 2. Montage showing the keyhole method of image acquisition. The pivot point is above the target anatomy, providing views of structures that would be hidden while using the inverted cone method. Case 1. This 54-year-old woman underwent a right orbitozygomatic approach to clip an aneurysm involving the right middle cerebral artery. The focus/pivot point for the QTVR image acquisition was the bifurcation of the CA. The images were acquired using the inverted cone method. The grid size was seven seven frames, yielding 49 images acquired over 20 minutes. Angulation was approximately 45 horizontally and vertically (Fig. 3). Case 2. This 70-year-old man underwent a transcallosal approach to resect a colloid cyst of the third ventricle. The pivot point was set to the level of the small craniotomy, and the images were acquired using the keyhole method. The grid size was six six frames, yielding 36 images acquired over 15 minutes. Angulation was approximately 45 horizontally and vertically (Fig. 4). Case 3. This 53-year-old man underwent a frontotemporal craniotomy to resect a mycotic aneurysm on the right distal middle cerebral artery on the cerebral surface. The pivot point for the QTVR image acquisition was the cerebral surface near the aneurysm. The grid size was seven seven frames, yielding 49 images acquired over 20 minutes. Angulation was approximately 45 horizontally and vertically (Fig. 5). Discussion Neurosurgeons routinely capture images during surgery. Most of these images are still photographs acquired with or without the aid of an operating microscope. The photographic and illustrative works of Harvey Cushing and Mildred Codding, 3 Dorcas Hager Padget and Walter Dandy, 6 Charles P. Hodge and Wilder Penfield, 10 and Albert L. Rhoton Jr. 17 serve as a testament to the efforts of neurosurgeons to communicate visually about their anatomical and surgical encounters in the operating room. Advances in photographic technology are increasing the number of images acquired in digital still formats. 15 Small video cameras that FIG. 3. Montage showing two enlarged images from within the grid system that reveal the surgical anatomy from different angles as the user pans horizontally or tilts up and down. The bifurcation of the right CA seen from a variety of angles reveals structures that may be hidden (for example, the CA bifurcation and its perforating vessels) in certain views. J. Neurosurg. / Volume 100 / April,

A. Balogh, et al. FIG. 5. Montage demonstrating that the lack of depth in the field (for example, when treating a lesion on the surface of the brain) provides no additional valuable information.

Montage demonstrating that, with the pivot point at the level of the craniotomy, the keyhole function allows images to be captured through a narrow corridor (for example, in a minicraniotomy or a

4 A. Balogh, et al. FIG. 5. Montage demonstrating that the lack of depth in the field (for example, when treating a lesion on the surface of the brain) provides no additional valuable information. Therefore, the standard QTVR mode is sufficient. FIG. 4. Montage demonstrating that, with the pivot point at the level of the craniotomy, the keyhole function allows images to be captured through a narrow corridor (for example, in a minicraniotomy or a transcallosal approach). The keyhole mode permits views of anatomical features that are obstructed and that otherwise would be visible only through a larger craniotomy. can mount on the operating microscope are also available. Although images may be played repetitively to review a case, the images or video sequences are limited by the restricted depth of field: only the target anatomy or lesion is in focus. Furthermore, the user cannot navigate through the image. The term QTVR refers to an image reconstruction method that is relatively independent of computer platforms. Multiple 2D or interlaced 2D images from an object that are acquired from slightly different angles are reconstructed into a single image montage by using commercially available software. The application of QTVR to medicine is quickly being recognized. 2,12,13,16 Interlacing of the images acquired from slightly different angles produces a sense of depth a stereoscopic effect. The stereoscopic QTVR system we have described allows the user to navigate through a depthenhanced montage with readily available, affordable computer hardware and software. Although the stereoscopic QTVR mode is not based on true 3D renderings, the image acquisition mode from two different eyepieces creates a stereoscopic display of the images. As a result, users can look behind or around structures in the field of interest by using a mouse or keyboard to pan, tilt, and rotate the reconstructed image. This system provides unique educational opportunities for its users, who can visualize target anatomical images acquired during live surgery within a more complete anatomical frame of reference than is offered by any other modality. With the aid of the stereoscopic shutter glasses, the intraoperative anatomy sequences can be readily combined for presentation or for review on standard personal computer workstations. If the user prefers it or lacks the hardware for stereoscopic viewing, the images can also be viewed in the standard QTVR format. The captured segments are easily distributed to storage media or embedded into internet web pages. We have used two methods to produce stereoscopic QTVR images. The first is produced around a fixed focal point. Conceptually, the focal point is like the tip of an inverted cone. This method is applied when the dissection environment is wide and deep (for example, orbitozygomatic exposure, far-lateral craniotomy, transfacial approaches, or approaches to the interpeduncular fossa associated with many small, important structures). The second method uses an external focal point located slightly above the bottom of the dissection (keyhole mode). As the viewer pans or tilts in one direction, a view of otherwise hidden structures can be achieved in the opposite direction by using the QTVR system (Fig. 6). This method especially affords views of structures that are otherwise partially obstructed or hidden and offers better views of the anatomy in narrower or restricted 594 J. Neurosurg. / Volume 100 / April, 2004

5 Intraoperative stereoscopic QuickTime Virtual Reality settings. Images obtained from the surface of the brain, as in Case 3, do not produce the striking 3D effect in the stereoscopic QTVR mode because depth of field is lacking. In such situations, the standard QTVR mode is a better option. Although the cameras, computer hardware and software, and visualization hardware (shutter glasses) are convenient and readily available, the robotic microscope system used as the platform is not. Few microscopes produced for the operating room are robotic or have automated control for such image acquisition. We use the model MKM microscope (Carl Zeiss Surgical, Inc.), which is well suited to this work because it can be controlled robotically and more precisely with specialized operational software. 2 The experience of seeing depictions of live anatomy acquired during surgery through a system producing stereoscopic images is exhilarating. The experience is further enhanced by the user s ability to define the orientation. Archived material combined with such image reconstruction has even greater heuristic value. Optimally, instead of splicing digital still images into a stereoscopic QTVR segment, images acquired with high-definition stereo video cameras while a continuous, smooth, automatic movement of the microscope is being executed would substantially decrease image acquisition time. This method would also provide an incredibly smooth interactive stereoscopic QTVR experience. Although there are indications that such virtual or semivirtual environment systems have educational benefits, most have not yet been widely evaluated. 1,7,19 The unanimous statements by our residents and fellows that they now have a better understanding of the anatomical relationships at least indicates that they enjoy lectures and presentations with immersive 3D content. Because we have based our software and hardware on widely available technology, 4 the educational value of such a system is amenable to evaluation. It will be incumbent on neurosurgeons to establish the educational value and benefit of these systems for training and continuing medical education. 5,8 The motivations are strong: opportunities for neurosurgical trainees to learn based on practical anatomy or the relatively unrestricted or legally unfettered surgical experience of previous generations may be waning. We believe that this and similar technology is workable and meaningful. Because neurosurgical trainees are restricted to 80-hour work weeks, along with subsequent maintenance of certification, this type of anatomical data display will be vital in both training and assessment of competency. Furthermore, impending legislation may restrict cadaver donations and substantially increase the already high cost of obtaining and maintaining cadaveric specimens. Such developments would adversely affect opportunities to practice surgical dissection in the laboratory. A fully simulated environment that mimics reality is still difficult to produce. 18 Most virtual systems require complex proprietary software or hardware. We have tended to remain within the realm of portraying real anatomy in reconstructed 3D or stereoscopic mode. Nevertheless, we are in the process of combining other technologies with stereoscopic imaging to obtain realistic representations of scenarios that cannot be seen using conventional photography or video recording. In the past 5 years, references to virtual systems have increased considerably. 18 We are combining J. Neurosurg. / Volume 100 / April, 2004 FIG. 6. Images demonstrating the two QTVR modes. A: The inverted cone method offers a limited field of view for image acquisition through a narrow surgical corridor. B: The keyhole method provides the entire field of view for image acquisition and 3D image reconstruction through the same narrow surgical corridor. This view would otherwise only be visible through a larger exposure. advances in computer technology, image processing, highfidelity rendering, haptics, sensors, and optics that may also lead to virtual neurosurgery for rehearsal and training purposes. Obtaining images for stereoscopic QTVR production in every surgical case by switching to a robotically controlled operating microscope is not feasible in today s cost- and time-conscious operating room. Nevertheless, creating an experience with depth, perspective, and user navigation from the real image content of an exquisite operation performed by a master surgeon offers a potentially valuable learning experience. Moreover, using cross-platform software and a standard computer workstation or laptop computer allows this imaging modality to become widely and conveniently available. 4 As the surgical microscope evolves and becomes a platform to which the surgeon may be connected only remotely (for example, becoming an imaging tool that delivers oriented images to the surgeon s goggles), advances will allow 3D or stereoscopic images to be captured in unique and increasingly practical ways. 14 Undoubtedly, various technologies will combine to produce fully immersive, real-time multisensory fusion of real and virtual information data streams into online, real-time visualizations recorded from actual neurosurgical procedures. 11,18 Such technology would permit 3D imaging of surgery for virtual reality-based education the creation of stereoscopic experiences that include the maneuvers of the surgeon or recordings made during the emergence of surgical problems that are expertly managed. Acknowledgments We thank Carl Zeiss Surgical, Inc., for providing the MKM operating microscope hardware and software support for this project. We also acknowledge Dr. Jeff Henn, a former resident at the Barrow Neurological Institute who is currently at the University of Florida in 595

6 A. Balogh, et al. Gainesville, for the original idea and development of microscopic stereo QTVR. References 1. Anastakis DJ, Regehr G, Reznick RK, et al: Assessment of technical skills transfer from the bench training model to the human model. Am J Surg 177: , Bernardo A, Preul MC, Zabramski JM, et al: A three-dimensional interactive virtual dissection model to simulate transpetrous surgical avenues. Neurosurgery 52: , Black PM, Moore MR, Rossitch E Jr: The Surgical Art of Harvey Cushing. Park Ridge, IL: American Association of Neurological Surgeons, Bullitt E, Aylward S, Bernard EJ Jr, et al: Computer-assisted visualization of arteriovenous malformations on the home personal computer. Neurosurgery 48: , Cusimano MD: Virtual reality surgery: neurosurgery and the contemporary landscape a three dimensional interactive virtual dissection model to simulate transpetrous surgical avenues. Neurosurgery 53: , 2003 (Letter) 6. Dandy WE: The brain, in Lewis D (ed): Practice of Surgery, Vol 12. Hagerstown, MD: WF Prior, 1932, pp Gorman PJ, Meier AH, Krummel TM: Simulation and virtual reality in surgical education: real or unreal? Arch Surg 134: , Gorman PJ, Meier AH, Rawn C, et al: The future of medical education is no longer blood and guts, it is bits and bytes. Am J Surg 180: , Henn JS, Lemole GM Jr, Ferreira MA, et al: Interactive stereoscopic virtual reality: a new tool for neurosurgical education. Technical note. J Neurosurg 96: , Hodge CP, Yamamoto YL, Feindel W: Fluorescein angiography of the brain the photographic procedure, J Biol Photogr 67:35 48, Hoffman H, Vu D: Virtual reality: teaching tool of the twenty-first century? Acad Med 72: , Kling-Petersen T, Pascher R, Rydmark M: 3D-brain 2.0 narrowing the gap between personal computers and high end workstations. Stud Health Technol Inform 50: , Kling-Petersen T, Pascher R, Rydmark M: Virtual reality on the web: the potentials of different methodologies and visualization techniques for scientific research and medical education. Stud Health Technol Inform 62: , Liu CY, Spicer M, Apuzzo ML: The genesis of neurosurgery and the evolution of the neurosurgical operative environment: Part II concepts for future development, 2003 and beyond. Neurosurgery 52:20 35, Melder PC, Mair EA: Endoscopic photography: digital or 35 mm? Arch Otolaryngol Head Neck Surg 129: , Nieder GL, Scott JN, Anderson MD: Using QuickTime virtual reality objects in computer-assisted instruction of gross anatomy: Yorick the VR skull. Clin Anat 13: , Rhoton AL Jr: The posterior cranial fossa: microsurgical anatomy and surgical approaches. Neurosurgery 47 (Suppl):S5 S6, Spicer MA, Apuzzo ML: Virtual reality surgery: neurosurgery and the contemporary landscape. Neurosurgery 52: , Wanzel KR, Hamstra SJ, Anastakis DJ, et al: Effect of visual-spatial ability on learning of spatially-complex surgical skills. Lancet 359: , 2002 (Letter) Manuscript received June 27, Accepted in final form December 15, Address reprint requests to: Robert F. Spetzler, M.D., c/o Neuroscience Publications, Barrow Neurological Institute, 350 West Thomas Road, Phoenix, Arizona neuropub@chw.edu. 596 J. Neurosurg. / Volume 100 / April, 2004

Interactive stereoscopic virtual reality: a new tool for neurosurgical education

Use your 3D glasses from the December 2001 issue to view images in this article. J Neurosurg 96:144 149, 2002 Interactive stereoscopic virtual reality: a new tool for neurosurgical education Technical