Term Paper Augmented Reality in surgery

Transcription

1 Universität Paderborn Fakultät für Elektrotechnik/ Informatik / Mathematik Term Paper Augmented Reality in surgery by Silke Geisen twister@upb.de

2 1. Introduction In the last 15 years the field of minimal invasive operation techniques has become, especially in surgery, more important [Sc04]. The research in this field exist since a long time, but it becomes more accepted and popular in the last years, because of the advantages for the patient. The biggest one is, that this technique reduces the possible trauma of a patient, which he could suffer from big scars. Another one is, that the postoperative recovery is most times faster. Because of this, minimal invasive operation techniques gives also better cosmetic results and is willingly used in plastic surgery. But besides the advantages there exist some problems [Fu98]. Given that the surgeon uses a camera for the operation, he has to look all the time on a screen above the operation table to know, where he is operating. Thus he makes small cuts and the look all time on a screen, the surgeon has got a limited field of view during the whole operation. Because of the camera the surgeon only sees what is directly in front of his operation scope. He has to readjust the camera all the time. On the one hand, this could do an assistant for the surgeon, but this needs good and much coordination. On the other hand, he could do it by himself, but this limits the surgeon that he can only operate with one hand. There is the possibility to use fixed cameras, but this is far to dangerous for the patient. The next problem is, that it is possible that the camera is not facing in the same direction as the surgeon. Because of this the movement of the instruments on the screen are not the same as the movement of the hands of the surgeon. It needs a lot of training for him to get familiar with the problem and to compensate it. Last there is the problem, that the imagery on the screen is only 2-dimensional. There exists a lack of depth information. He could estimate the distance of the specific structures by moving the camera. To improve the actions of the surgeon during the operation there are stereo endoscopes reported. But there is still the first problem, the limited field of view, because he could only see what is directly in front of the camera. A possible resolution for this problems is the use of Augmented Reality in combination with the operation techniques. 2. Augmented Reality Augmented Reality, short AR, is a mixture of the reality and the virtual reality. It is an environment, which includes elements of both, virtual objects and real-world elements. Sometimes Augmented Reality is also called a mixed reality. In the AR the virtual objects overlay the reality. In his survey, Azuma says more concrete, that Augmented Reality (AR) is a variation of Virtual Environments (VE), or Virtual Reality as it is more commonly called. VE technologies completely immerse a user inside a synthetic environment. While immersed, the user cannot see the real world around him. In contrast, AR allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. Therefore, AR supplements reality, rather than completely replacing it. Ideally, it would appear to the user that the virtual and real objects coexisted in the same space. [Az97, 1.2 Definition]. In Figure 1 a picture of an Online Learning Mode of the project Vampire is shown [Ba04]. This project works with a combination of Augmented Reality, pointing gestures and speech recognition (system ESMERALDA). The picture shows a real rubber and sharpener on a real desk. These are overlayed with their labels in the virtual environment. Additionally on the right side is a virtual menu, which can be used. Now, the user points with his finger at a real object. The AR system recognizes the pointing gesture and calculates, where the user is pointing. If the object which is pointed at is recognized, the virtual object is blended in and the system highlights the object. Here, a cup is highlighted, but it has been falsely classified as Smint. With speech recognition, the cup can be new classified with the right label. 2

3 Figure 1: Vampire Online Learning Mode Augmented Reality systems often involve the use of a Head Mounted Display. There are two main categories of HMD's based in the AR environment [In04]. First there are Optical See-Through HMD's which work by placing optical combiners in front of the user's eyes. Theses combiners work like a semitransparent mirror. They allow the user to see through them. On miniature displays in the line of sight of the person the virtual objects are portrayed with a reflection of the combiners. Figure2: Scheme of a OST-HMD Figure 3: An Optical See-Through HMD Second, there are Video See-Through HMD's, which uses miniature cameras to capture the view of the world, that would be seen by the eyes. These video images of the real world are now combined with the virtual objects. The combining of the virtual and the real world images can be achieved during two different methods, depending on the depth information. The first method is to use a blue screen approach, where regardless of the depth, the results of the approach overlay the real world images. A better method is to perform a depth-composite of the real and virtual images. But this requires a depth information of each pixel in the real world image. 3

4 Figure 4: Scheme of VST-HMD Figure 5: A Video See-Through HMD Another possibility for Augmented Reality systems is that they involve the use of a PDA or a cell phone. This is called Hand-held Augmented Reality. Examples for this are the KickReal engine which was developed in the C-Lab in Paderborn [kic] or the research projects of the Bauhaus university in Weimar [wei]. Figure 6: Hand-held Augmented Reality For the tracking and blending in of virtual objects it often involves the use of special markers. These markers are placed on a specific position. If the markers are tracked, the object will be blended in directly on the marker or on a position which is specially defined in acquisition to him. The other possibility of tracking is to define special forms, for example the foot, as it is done in the project KickReal. Figure 7: Kanji marker 4

5 There are several examples of applications where Augmented Reality could be useful and is in use. In the industry field, especially for repairing machinery. Generating of special 3-dimensional maps could be used in the military area. One big field for AR applications are games, such as KickReal, a soccer game for the cell phone. A virtual ball can be shoot with the real foot, captured by the camera of the cell phone, in a virtual goal. Or there exists an Augmented Reality version of the game Quake called ARQuake in Australia. One of the newest invention is a version of the game Pacman called Human Pacman for AR. Last but not least, AR applications could be useful in medicine, especially surgery and accordingly for minimal invasive operation techniques. 3. Requirements for surgery First, the top priority in the medicine field is that there is the highest precision required on systems. If a system is not exact, it could be very dangerous for the patient and he could take irreparable damage, if the surgeon makes a mistake. To use an Augmented Reality system, especially in surgery, there are several special requirements. For using AR a Head Mounted Display or an AR-display is needed. For operations, the Optical See- Through HMD has to be favoured over the Video See-Through. The surgeon is able to see the whole scenery all the time. But even if the OST-HMD has such advantages, there are the problem that the surgeon is often limited in his liberty of action. Secondary, there is the problem to make the HMD sterile. At the moment, this is not really possible and because of this, an HMD could not be used [Sc02]. To solve the problem of sterility, an AR-display as it is used in the medarpa system in Darmstadt, could be used. It works similar to the optical see-through HMD, not as semitransparent mirror, but more as a semitransparent window [Sc04]. The surgeon is able to see the patient through the display and the 3-dimensional model of the patient is blended in at the same time. Figure 8: AR-Display of the medarpa system in Darmstadt The AR-display of the medarpa system is shown in figure 5 [med]. It shows the hand of the surgeon under the display with the operating instrument in his hand. The instrument is highlighted, especially the tip of it. This is important for the localization of the instrument during the operation. In the picture the surgeon is operating on a phantom, not on a real patient. This was for a demonstration. Next the use of an Augmented Reality system for surgery requires some special markers. These are important for the tracking during the operation and for the registration. These markers could be similar to band-aids or are with special fluorescent colour. They are sticked on the patient and for an operation it requires five markers at minimum. It is important to have exact algorithms for the tracking of the markers, which are often optical tracking systems, such es e.g. EOS. There are also needed such algorithms for the tracking of the operating instruments. Most times for the operating instruments the tracking algorithms are electromagnetic. Additionally to these tracking algorithms it requires navigation support for the instruments, because the transparent display does not offer a stereoscopic view to the surgeon. To compensate this problem and to know every time, where the instrument list, there is a special colour feedback 5

6 required. Some developed programs use for this task the virtual colour of the instrument e.g. in a colour sequence like traffic lights. If the instrument is pointing in any direction, the colour of it is red. When it is pointing in the direction of the operating target, the colour changes to yellow. The surgeon is now able to move the instrument directly to the target, if he takes care, that the colour is always yellow. If he reaches the target region the virtual colour of the instrument changes to green. There also other methods to use for navigation and there exists special software, like ARION (Augmented Reality for Intra-Operative Navigation). To generate an exact 3-dimensional model of the patient from the recordings of the CT or MRT, very good and exact visualization tools are required. The AR system which is used in surgery has to be very efficient. There also has to be a very high real scene resolution. The surgeon must have a good view of the real scene of the OR and on the patient. This is achieved by the AR-display. Even if there are some cut backs because of the reflection of the light on the display and the transparency, the surgeons, who experimented with it, thinks they are more than good enough for such a big task as operating. To use the system in intra operative operations, it has to be in real time quality and safe. In this context it is required to achieve a high accuracy in tracking in registration. The registration is estimated to be under 1 cm or less. In the end, such a system hast to be fast, very exact, easy to use for the surgeon and robust. It has also to be relatively inexpensive that the hospitals and at the end the patient can afford it. 4. Interaction Before starting to operate the patient, there is some previous work to do. There have to be taken recordings of the patient such as CT's, MRT's or ultrasonic images. Most times CT's are used. The images are taken in conjunction with markers for AR. This is important for the registration during the operation. For this, the special markers are attached to the patient's body. These markers are attached before the patient is scanned and their position is the same in the later virtual 3- dimensional model of the patient After all marker positions are recorded, the virtual patient can be registered to the physical patient by registering two 3D point sets. The CT ore some other required procedure will occur with special radiopaque material. The patient has to be in the same position later in the operation as during the CT. With these recordings will be generated a 3-dimensional model of the patient, which is registered to the physical patient as said before and it overlays him through the operation [med]. Before the operation starts, the patient he has to be under anesthetic and placed in the same position on the operating table as in the CT. After that the surgeon covers the AR-display with a sterile plastic film all over to make it antiseptic. Because of this, he is able to move the display during the operation by himself in every direction he wants. At the beginning he moves the display all over the patient to get an overview of his anatomy. On the basis of the markers the tracking system balances them with the real patient and the virtual patient. The 3-dimensional model of him appears in the AR view and the surgeon gets a view inside the patient. After that the surgeon places the display over the region to operate, where the region appears also as the 3-dimensional model on the display. As the surgeon starts to operate, the instruments and his line of sight is tracked. Because of this, he and the system know every time, where are the instruments. At the moment, where the instrument contacts the skin, a shiny point appears in the AR view at the tip of it. This is important to for the localization of the instrument on the one hand and to localise the target region. A projection of a virtual axis inside the body appears as well. With the movement of the instrument not only the direction to enter the body, but the angle to enter could be defined. If the surgeon finds the best point to make the cut and to enter the body of the patient, the virtual axis is fixed and a straight line will be projected inside him. After the entry point is fixed, the rest of the operation could be executed through him, like enter the biopsy needle, insert a probe for later 6

7 radiation or to apply a single bypass. Examples for the use of such a system in surgery could be the heart surgery, especially to apply bypasses. Or it could be used in the liver surgery, pneumology and radiooncology as well. Also it could be useful for biopsies. Figure 9: Set-Up of an AR-System [Sc04] During the research and experiments there occur some problems and disturbances of the system [Sc04]. One great problem is the navigation error and means the accuracy of the navigation of th instruments. This is the error between the achieved target and the desired target. There were some good results, where the error was nearly null, but there were also results, where the error was over 10 mm and that is in surgery far too much. This problem is influenced by diverse accuracies. First there is to mention the registration. If this is not exact, there could be great differences between the seen 3-dimensional model and the physical body of the patient. Another reason could be, that the calibration of the instruments is not accurate. Or the position or rotation of the sensor of the instrument are not perfectly. If all these are not working exact and the navigation error is to high, in a real operation could this mean irreparable damage to the patient in the worst case. Another experience was, that there could be observed major disturbances especially of the electromagnetic tracking system near CT's and high voltage cables. The best would be to use the system in an undisturbed environment. Figure 10: Set-Up of the Medarpa system 7

8 5. Conclusion As it was seen, the use of Augmented Reality systems in combination with surgery could be of much help, especially in the minimal invasive operation techniques. The AR would solve the problems, which were mentioned at the beginning. The surgeon has got a good field of view during the operation. He is not staring most of the time on a screen, but through a window on the patient himself. This also means, he has no lack of hand-eye coordination, because he sees all the time the movement of his hands. There is also the problem of the depth problem of 2-dimensional imagery solved, because the surgeon view the 3-dimensional model of the patient on the AR-display. With this, he has a view inside the patient. So there is a minor puncturing of the skin, which means small scars. That indicates, the patient could be prevented from a trauma of a big scar and the healing process would be much faster. This makes the system also a technique for the plastic surgery. The operation could be made faster and first of all exacter. It could be avoided the puncturing of important organs. If the operation would be exacter and faster, that could mean, that the patient will be shorter under anaesthetic. In the end this would be mean lesser costs for the hospital, for the health fund and in the end for the patient himself. Unfortunately at the moment Augmented Reality systems are only used in research for surgery or as training modules for new surgeons. The future work would be to minimize the navigation error and to make visualization tools, registration and tracking more exact. The possible use of Augmented Reality systems as regular use in surgery is expected as five to ten years. There exists some projects which use AR systems for training for new surgeons, e.g. ARISER in Norway [ari]. Another very interesting project is ARES (Augmented Reality in Enhanced Surgery) of ircad in Strasbourg. They are operating on tumors. They made very good experiences with the use of an AR system, they achieve their target under 2mm. Because of this they gave the statement on their homepage, that they now wanted to validate the system on real patients [are]. One idea for the future could be tele operations. There the surgeon is not standing in the operating room, but elsewhere. He is wearing a Video See-Through Head Mounted Display. He sees a video of the scene in the operating room, as he would be standing in it by himself. To accomplish the operation he would be using haptic hardware. But this is at the moment only a suggestion and for the far future. Figure 10: ARES, external view of patient in virtual transparency Figure 11: ARES system to see the patient in transparency 8

9 6. References [are] Strassburg, last visited: [ari] Norway, last visited: [Az97] [Ba04] [Fu98] [In04] A Survey of Augmented Reality, Azuma, Ronald T., Teleoperators and Virtual Environments 6, S , Malibu, August 1997 Visual Learning of Motion Behaviours and Classification of Spatiotemporal Events, Bax et. Al, Report, Bielefeld, Juli 2004 Augmented Reality Visualization for Laparoscopic Surgery, Fuchs et al., MICCAI'98, 1998 Mixed and Augmented Reality, Interrante, Victoria, Institute of Technology Labs, presentation at class on , university of Minnesota [kic] last visited: [med] Darmstadt, last visited [Sc02] [Sc04] [Su02] Medarpa Ein Augmented Reality System für Minimal-Invasive Interventionen, Schnaider et. Al, Virtuelle und Erweiterte Realität, Leipzig, November 2002 Implementation and Evaluation of an Augmented Reality System Supporting Minimal Invasive Interventions, Schwald et. al, AMI-ARCS 2004, France, September 2004 A concept work for Augmented Reality visualisation based on a medical application in liver surgery, Suthau et. Al, IAPRS, Corfu, 2002 [wei] h ttp:// last visited: