A Panoramic Wireless Endoscope System Design for the Application of Minimally Invasive Surgery

Transcription

1 Vol. 2, No. 2, pp (2014) A Panoramic Wireless Endoscope System Design for the Application of Minimally Invasive Surgery Chun-Hsiang Peng 1 and Ching-Hwa Cheng 1,* 1 Department of Electronic Engineering, Feng Chia University, Taichung, Taiwan, ROC * Corresponding Author / d @gmail.com KEYWORDS : Minimally invasive surgery, Wide viewing angle mosaicing, Wireless image transmission Minimally Invasive Surgery (MIS) is the current trend in surgery. Compared to traditional surgery, MIS can substantially decrease recovery time and expenses needed by patients after surgeries, reduce pain during surgical procedures, and is highly regarded by physicians and patients. An endoscope is widely used in the diagnosis and treatments of various medical disciplines, such as hysteroscopy, laparoscopy, and colonoscopy, and have been adopted by many branches of medicine. However, the limited image field of MIS is often the most difficult obstacles faced by surgeons and medical students, especially to less experienced physicians and difficult surgical procedures; the limited field of view of endoscopic imaging does not provide a whole picture of the surgery area, making the procedures difficult and full of uncertainty. In light of this problem, we proposed a "Panoramic Wireless Endoscope System design", hoping to provide physicians with a wide field of view of the endoscopic image. We combine images captured from two parallel-mounted endoscope lenses into a single, wide-angle image, giving physicians a wider field of view and easier access to the surgical area. In addition, we developed a wireless transmission system so the image can be transmitted to various display platforms, eliminating the needs for excessive cabling on surgical tools and enable physicians to better operate on the patient. Finally, our system allows surgical assistants a better view of the operation process, and enables other physicians and nurses to remotely observe the process. Our experiment results have shown that we can increase the image to 152% of its original size. We used the PandaBoard ES platform with an ARM9 processor and 1G of onboard RAM, and continuously implementing animal trials to verify the reliability of our system. Manuscript received: March 10, 2014 / Accepted: March 18, Introduction The endoscope is an essential accompanying medical equipment of the Minimally Invasive Surgery (MIS). MIS refers to the act of making a small surgical incision to insert a surgical camera lens (laparoscopes or thoracoscopes are commonly used) and other instruments to perform surgical operation. This technology is gradually replacing traditional surgical operations, due to the ability to reduce wound openings, patient discomfort (wound pain), and prevent unsightly surgical scars. Because of its minimally invasive nature, patients recover much quicker and hospital stay is lessened. However, MIS is usually conducted in a narrow field of view with small operating space, and often involves delicate tissues that require greater skills and results in much higher difficulty than traditional surgeries. The advent of MIS has resulted in the demand for greater image clarity and smaller device volume of endoscopic imaging, which in turn drives forward the demand for integrated circuit medical devices. When comparing the changes to physicians brought forth by MIS to the traditional surgery, the biggest difference lies in the methods of observation and operation. Traditionally, surgeons are able to directly observe the patient's condition. Taking abdominal surgery as an example, doctors can directly view the operating area, have a wider field of view and viewing angle, and have tactile feedback during operation. In the case of minimally invasive surgery, since the biggest draw is opening wounds small enough for only the instruments to go through, doctors are not able to directly view the operating area, but rather operate through viewing the internal images of the abdomen, transmitted to a flat screen display through an endoscope. Limited by the narrow field of vision of the endoscope, doctors have a difficult time grasping the full picture of the operating area; and since the display only provides information in two-dimensions, it is also difficult to pinpoint relative positional relationship of the surgical instruments to each organs. Therefore, physicians require extended periods of training to be familiar with MIS operations and visual representation, a feat that is often difficult to complete by younger and less-experienced physicians. In the field of computer graphics or imaging processing, the method of combining multiple over-lapping images into a larger image is known as mosaicing. This technique has been implemented in medicine such as retinal image [1]. There are also a few initial researches done in the field of minimally invasive surgery, such as 91

2 Vol. 2, No. 2, pp (2014) those from Behrens et al [4], Miranda-Luna et al [3] and Behrens [2], who demonstrated endoscoping mosaicing of bladder images; [4] and [3] attempted to conduct continuous image mosaicing during the surgical operation on bladder cancer, with the endoscopic imaging filming the process of entering bladder from the urethra. Reeff et al. [5] combined the endoscopic images of the placenta. The purpose of compositing fetoscopic images is to provide the doctor a wider field of view to measure the abnormal vascular connections in the placenta of twin-to-twin transfusion syndrome [6]. discussed about the possibility of dynamic non-optical endoscopic image expansion for the use of sinus surgery. The literature mentioned the use of a desktop computer to achieve double image expansion. Loewke et al. [7] proposed the use of surgical robot to achieve medical image mosaicing, catering to micro-endoscopic application of micron-scale image resolution in very narrow field of view. The literature demonstrated image mosaicing technique to provide real-time macrosized images. In [8], image mosaicing was applied to endoscopic coronary bypass surgery. The literature has shown that image mosaicing has been gradually applied in the field of medicine, however such application is still limited to compositing of small fields of views, such as blood vessels and urethras, although they do indirectly demonstrated the feasibility and assistive property of clinical applications of image mosaicing. To this date, a real-time panoramic endoscope has not yet been published. Unlike the currently medical image mosaicing technique [1], the image-based panoramic expansion shown in Fig. 1 is the mapping of the internals with images captured from the path of the endoscope. The image we are seeing is actually the real-time endoscopic image viewed by the physician. Fig. 1 Panoramic image expansion and mosaicing [1] 2. Functions and Concepts As the feature points of medical images are generally different from normal points, if we are able to develop a multi-angle, wirelessly transmitted small endoscope, and install it on Trocars of various positions, we could obtain wider angle, real-time images, and obtain the relative positions through calibrations of multiple lenses, and rapidly composite into a dynamic, multi-angle and wide-angle image. Requirement for this technology includes small tube dimensions and image clarity; having a smaller tube dimension will reduce patient's pain during examination. We propose a panoramic wireless endoscopic system design that can be used in surgery. We conceived a Trocar integrated and wirelessly transmitted internal micro wide-angle endoscope. The system schematic is shown in Fig. 2. Images are captured through multiple lenses to determine their degree of variations, and we used different algorithms to implement image mosaicing and reduce power usage. The images are then encoded and compressed for wireless transmission outside of the body, and are received and decoded externally to display the captured image data. Fig. 2 System schematic Designing an internal wide-angle, multi-lens wireless endoscope requires the combination various core technologies such as image mosaicing, compression, decompression and wireless transmission. The endoscopic image processing of minimally invasive surgery incorporates the following key issues: (1) Multiple lens image composition to provide panoramic image (2) High speed wireless transmission of image data (3) Image compression and decompression (4) Cross-platform technology 3. Wide-angle image compositing technology To increase the viewing angle of the endoscope, we used multiple sets of lenses to implement image mosaicing, expanding the physician's field of view. Wide-angle compositing is comprised of two parts: Image Registration and Image Compositing. The flow diagram is shown in Fig. 3 and the process is as follows: In this step, the relationship between two image sources are explored and investigated. This is the most important step of the whole compositing process, as it directly influences the accuracy of the resultant image. We first searched for the feature points between two images, and match the feature points, then established a linear 92

3 Vol. 2, No. 2, pp (2014) relationship between the two images based on the matching result, shown as Fig.4. Input Images Image Compositing Warp images Compositing images avoid impeding the surgical process. Thus, we have chosen to use the RTSP streaming technology. Fig. 5 shows the schematic of RTSP streaming. The client end (WINDOWS, IOS, Android) requests the server end (DVR) to conduct effective multimedia data transmission in networks with limited bandwidths. Failure Image Registration Find feature Matching feature Blend images Output Image Enough match? True Fig. 3 Image compositing process Fig. 5 RTSP streaming framework Fig. 4 Final wide-angle composited image 4. Wireless Transmission Technology We used the Linux system to set up a RTSP (Real Time Streaming Protocol) protocol for cloud-based, cross-platform network play-back and management. As the file size of the images is larger than normal files, they can only be played after they have been downloaded completely. To enable real-time image playback, we employed streaming technology. The framework of multimedia streaming includes the following modules: media source, compression, streaming server and player software. The compression module's main function is to compress the large medial files into manageable sizes that may be rapidly transmitted over the network. It also needs to divide the compressed media into packets for network transmission. The H.264 codec has high compression efficiency that maintains higher quality under the same compression ratio. The image data is compressed and stored; compressed video streaming during transmission will reduce the display information to enable synchronization. The streaming server acts to establish and manage streaming and transmission services. The player software receives the data packets from the server, decodes and playback the media data. The table shows that RTP streaming is faster than HTTP streaming, which is suitable for use in MIS, where real-time image display is required During surgery, the monitor displayed image will need to match with the actual situation to eliminate misjudgments by the physician. The RTSP protocol utilizes the UDP communication layer during data transmission, which reduces the data volume needed during transmission, decreases the transmission latency, improves the consistency of the images, and provide doctors optimal surgical conditions. We used DVR and wireless routers to build the transmission system, and we may request the DVR for real-time playback of videos on platforms such as Windows, IOS and Android. 5. Experimental Results We used the development platform of PandaBoard ES, and wrote the software in C with image functions from OpenCV We first conducted the search for feature points, and as shown in Fig. 6, the top figure shows the left image and the obtained feature points; the bottom figure shows the right image and the obtained feature points. The figure showed that feature points are more easily obtained on the edges of images and bright sections. The result of feature point matching is shown in Fig. 7. Although there were a few errors in the matching process, as long as the required match is correct, the linear relationship between the two images can be established. To confirm the stability of our system, we also participated in animal trials and conducted image mosaicing with endoscopes. The results are shown in Fig. 8. Our data showed that with our method, we can increase the image 55.2% times its original size. In this study, we employed wide-angle compositing technique to expand the field of views of endoscopic images. Our tests have 93

4 Vol. 2, No. 2, pp (2014) shown that we were able to increase up to 55.2% of the originalimage size, and we also conducted animal trials to verify the feasibility of our system (as shown in Fig. 9). We plan to optimize our algorithm in the future to enable dynamic and real-time display of the images, and to develop the system into an IC chip, which allows us to build a low power, wireless and wideangle endoscope, such as the system depicted in Fig. 2. Our cross-platform image transmission system can be used to play back images on mobile phones, tablet devices and computers, which facilitates in learning and assistance by assistants during surgical procedures. Fig. 10, Fig. 11 and Fig. 12 depict the actual cross-platform system. There are current two sets of endoscope mock-ups; one set incorporates the endoscope into the MIS instruments to decrease the size of the tools; the other set places the lens on the both sides of the Trocar, providing left and right images. Based on the previous criteria, our wide-angle wireless endoscope should incorporate the following features: (1) Small footprint, battery operable (2) Wireless to allow freedom of movement; does not takes up operating space of the surgical instruments (3) Temperature controllable to prevent the endoscopic equipment from damaging the body tissues (4) Real-time, wide viewing angle and clear image Left image Left image Animal trial result 2 Right image Right image Fig. 8 Animal trial results Fig. 6 Obtained feature points Fig. 9 Participation in animal trials Fig. 7 Results of feature points comparison Animal trial result 1 Fig. 10 System configuration of the transmission end 94

5 Vol. 2, No. 2, pp (2014) Fig. 11 System transmission of the receiving end Fig. 12 Application on smart phones Our system is targeted toward the application of wireless endoscope used in minimally invasive abdominal surgery. Due to its requirements for a small footprint, multiple lenses for image capture and wireless transmission of compressed image data, a lower power design is needed to extend the battery operation time. The technology we have devised will be integrated into future wireless, multi-lens endoscopic systems, but in the current study, we have focused on the design of image expansion and wireless transmission. ACKNOWLEDGEMENT This paper was supported by the Ministry of Education s subsidy to Smart Electronic Inter-campus Federation. The project title is Healthcare Medical System and Biomedical Image Processing System. We thank the support from the Ministry of Education for implementation of this project. [1] A. Can, C. V. Stewart, B. Roysam, and H. L. Tanenbaum, A feature-based technique for joint, linear estimation of highorder image-to-mosaic transformations: mosaicing the curved human retina IEEE Transactions on Pattern Analysis and Machine Intelligence, 24, (2002) DOI: / [2] A. Behrens, Creating Panoramic Images for Bladder Fluorescence Endoscopy Acta Polytechnica, 48, (2008) [3] R. Miranda-Luna, C. Daul, W. C. P. M. Blondel, Y. Hernandez- Mier, D. Wolf, and F. Guillemin, Mosaicing of Bladder Endoscopic Image Sequences: Distortion Calibration and Registration Algorithm IEEE Transactions on Biomedical Engineering, 55, (2008) DOI: /TBME [4] A. Behrens, T. Stehle, S. Gross, and T. Aach, Local and global panoramic imaging for fluorescence bladder endoscopy, Proc. Annual International Conference of the IEEE. Engineering in Medicine and Biology Society (EMBC 09), pp , September (2009) DOI: /IEMBS [5] M. Reeff, F. Gerhard, P. Cattin, and G. Székely, Mosaicing of endoscopic placenta images GI Jahrestagung, 93, (2006) [6] Y. Yamauchi, Non-optical expansion of field-of-view of the rigid endoscope, Proc. World Congress. Medical Physics and Biomedical Engineering 2006, pp , January (2007) DOI: / _1061 [7] K. E. Loewke, D. B. Camarillo, C. A. Jobst, and J. K. Salisbury, Real-time image mosaicing for medical applications Medicine Meets Virtual Reality, 15, (2007) [8] M. Hu, G. Penney, D. Rueckert, P. Edwards, F. Bello, M. Figl, R. Casula, Y. Cen, J. Liu, Z. Miao, and D. Hawkes, A Robust Mosaicing Method with Super-Resolution for Optical Medical Images Medical Imaging and Augmented Reality (MIAR 10), Springer Berlin Heidelberg, pp , September (2010) DOI: / _39 [9] C. Harris, M. Stephens, A Combined Corner and edge Detector, Proc. of Fourth Alvey Vision Conference, pp (1988) DOI: /C.2.23 [10] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, Speeded-Up Robust Features (SURF), Computer Vision and Image Understanding (CVIU), pp , September (2008) DOI: /j.cviu [11] M. Brown, D. G. Lowe, Automatic Panoramic Image Stitching using Invariant Features International Journal of Computer Vision, 74, (2007) DOI: /s [12] P. J. Burt, E. H. Adelson, A Multiresolution Spline With Application to Image Mosaics ACM Transactions on Graphics, 2, (1983) DOI: / REFERENCES 95