Influence of visual position information in a tissue stiffness discrimination task with haptic feedback

Transcription

1 Workshop On Virtual Reality Interaction and Physical Simulation (2005) F. Ganovelli and C. Mendoza (Editors) Influence of visual position information in a tissue stiffness discrimination task with haptic feedback O. Tonet, G. Megali, and P. Dario CRIM, Scuola Superiore Sant Anna, Pisa, Italy Abstract Minimally invasive surgery has become a widespread alternative to open surgery, thanks to robotic and mechatronic instruments, capable of restoring part of the visual and tactile sensory information lost due to the small accesses. Relative importance of haptic and visual information in interfaces for teleoperated surgery is a debated issue. This paper describes a study about the influence of visual position information on the capability of subjects to discriminate tissue stiffness. The subjects, by pinch grasp palpation through a haptic interface, had to discriminate between tissues as the difference of their stiffness decreased. For each subject, the same experiments were carried out both in presence of only haptic feedback, providing force information, and with added visual feedback, providing also displacement information. Subjects scored significantly better when also visual feedback was present, rather than with only haptic feedback. We conclude that visual position information plays an important role in haptic stiffness discrimination tasks. Categories and Subject Descriptors (according to ACM CCS): H.5.2 [User Interfaces]: Haptic I/O 1. Introduction In the last few years, minimally invasive surgery (MIS) is becoming a more practiced and convenient alternative to open surgery, thanks to the introduction of novel robotic and mechatronic surgical instruments, which restore part of the visual and tactile sensory information lost due to the small accesses. Indeed, during MIS procedures, the perception of the surgeon of the anatomical regions involved in the operation is less effective, as in the case of vision, or even absent or distorted, as in the case of tactile and haptic perception. In this context, it is clear that the partnership between engineering, computer science and medicine, represented by computer-assisted (or -integrated) surgery (CAS or CIS) [TLBM96], can provide very powerful means for the development of a novel generation of surgical instruments. Research efforts are currently devoted to enhance the dexterity and sensory capabilities of surgical instrumentation for MIS, to improve visual and tactile rendering to the surgeon, and to develop complete systems able to integrate multimodal data (images, models, planned procedures, sensor signals). The relative importance of haptic and visual information in MIS surgery procedures is a most debated issue, though the arguments are sometimes heavily influenced by personal opinions and feelings. The discussion is especially lively in the case of video-assisted surgery, where the major source of information about the operating site is represented by the endoscope image, whereas tactile information is almost completely lost and force (haptic) feedback is filtered by the presence of mechanical constraints (the access points), friction (between instrument and trocar), and backlash between instrument components. In traditional endoscopic surgery performed with manual instruments, the attenuated haptic sensation still plays an important role [PBF97, SD97, BHM 99], but it cannot be further restored or enhanced. In the case of mechatronic or robotic instruments, however, the interaction forces and torques between instrument and tissue can be measured and graphically displayed on a screen [DCM 00], using the so-called sensory substitution technique, or, more intuitively, rendered to the surgeon s hand as real forces and torques by means of actuators [RHMS99] and dedicated interfaces [Bur00]. Many previous works analyze the advantages that derive from the introduction of haptic feedback in

2 virtual environments for simulation [LVJ 04, BDK 04] and robots for teleoperation [WSH02, NMK04], where traditionally only visual rendering is present. Several works have measured forces involved in MIS procedures [RHMS99, KOB 02, ÇWTS01]. Recent surgical instruments [RHMS99, TPM05] and telerobotic systems used for research [MNK 05, ÇWTS01] incorporate haptic interfaces. Also, efforts for sensory substitution for rendering force and touch information by means of visual and auditory cues have been performed [KDOY05, Mas95]. Most of the debate is focused on whether and how much haptics can enhance visual feedback coming from virtual reality (VR) or video images. There is another branch of MIS where haptic feedback is naturally present and important, and visual feedback about instrument position is either absent or of minor importance than in endoscopy. An example is given by many percutaneous procedures such as biopsy or hyperthermia procedures [HRH 04]. In these procedures the position of the device tip, be it a needle, antenna or catheter (we will use the generic word probe throughout this paper), is measured with various medical imaging devices or localizers. Position information is obviously indispensable for navigation, but a debated issue is whether visual position information is useful, in addition to proprioception, for enhancing the feel of mechanical properties of tissues and surfaces, such as stiffness and textures [LA81, KLM93]. Visual position information, if proved to be useful in applications where traditionally only haptic feedback is present, could be provided separately as an enhancement by means of VR. This paper investigates the added value of visual feedback over haptic feedback. In particular we want to determine whether visual feedback about the position of the probe does enhance the haptic force perception in a tissue discrimination task. Similar works have been carried out in teleoperated needle insertion [GMO02, KH95]. Aim of this paper is to assess the importance of visual position feedback in haptic stiffness discrimination tasks. The following section explains the experimental setup and protocol, together with the data analysis techniques. Subsequently, results are presented and discussed. 2. Methods 2.1. Experimental setup Experiments were performed to assess the influence of visual feedback in a haptic tissue discrimination task. The architecture of the experimental setup is depicted in Figure 1. The force feedback is provided to the subject by means of a haptic interface (PHANTOM Premium 1.0, SensAble Technologies, Inc., Woburn, MA, USA), visual feedback is provided by means of a graphical user interface (GUI). The haptic and visual feedback loops run on two separate PCs: the haptics loop was written in C++ using the GHOST 2.1 haptics library and runs on a Windows NT 4.0 workstation. The PHANTOM is connected to the workstation by means of a dedicated PCI board. The visual loop was written in C++ using QT 3.2 GUI library and runs on a Windows XP workstation. Simulation of the tissue is performed on the graphics workstation. Intercommunication is performed by means of TCP/IP client/server modules based on MFC CSockets. Simulation of the tissue is performed by the graphics workstation, which reads the position of the subject s finger from the server, computes the corresponding force and sends the force to the haptics workstation, which feeds the force back through the haptic interface. Position of the subject s finger and the corresponding tissue deformation can also be shown on the GUI, as depicted in Figure 2. For this study, we chose to implement a simple tissue simulation and visual representation method, to allow users to better concentrate merely on force and displacement information. The visual feedback rate is at full monitor refresh rate (100 Hz); the haptic feedback loop runs at about 1 khz on the haptics workstation. However, since the tissue is simulated on the graphics workstation, due to the network communication, the intensity of force feedback is updated at about 130 Hz. Figure 2: The graphical user interface used in the experiments. The widget on the left simulates graphically the position of the probe and the corresponding tissue deformation. During the experiments, subjects have been asked to probe the stiffness of simulated tissue samples with their index finger. They wore the PHANTOM thimble on the index finger. The subject s hand was positioned in pinch grasp position, with the ball of the thumb resting on a vertical frame. In order to simulate the tissues on the haptic interface, we built a program that constrained the PHANTOM motion on a short horizontal line segment of 20 mm length. By closing the pinch grasp, the subject felt the force on the index pulp as he/she was grasping the tissue between index and thumb.

3 Figure 1: Schematic representation of the system architecture: two workstations are used, one for the visual feedback loop and one for the haptic feedback loop. Simulation is performed on the graphics workstation. The workstations are connected through ethernet networking Tissue models A realistic scenario, in which finest tissue discrimination capability is required, is represented by fetal heart surgery, e.g. in the case of the removal of the pulmonary valve obstruction by means of a radio frequency catheter in pulmonary atresia [STKT 01]. In this type of intervention, haptic feedback is absent and the procedure has been performed only by means of image-guidance. However, the use of microsensors at the distal end of the catheter could provide intraoperative measurement of the tissue stiffness [MECD03]. The stress/strain curves used during the experiments were derived from in vitro measurements on tiny samples dissected from the heart of a fetal lamb. In particular, endocardium wall and pulmonary valve samples were used. The stress-strain curves were measured with the apparatus described in [ETM edb]. The stiffness of the endocardium wall and pulmonary valve samples is very different and in an experiment similar to the one described here, where however only haptics feedback was involved, the subjects could very easily discriminate between the two tissue types [ETM eda]. Therefore we decided to interpolate intermediate stress/strain curves. Since hysteresis between the load-unload phases of the in vitro measurements was very small, and negligible for the tissue discrimination experiments (see experimental design in [KRS03]), we can describe the force/displacement curves as F = f v(x) for the pulmonary valve tissue and F = f w(x) for the endocardium wall tissue. In both equations F represents the force measured for a given displacement x of the probe. We linearly interpolated these curves by computing F = [ f v +t( f w f v)](x) that, with t [0, 1], provides intermediate force/displacement curves. In particular, t = 0 selects f v and t = 1 selects f w Experimental protocol We used the interpolated curves that simulate tissues with different stiffnesses as stimuli for the haptic/visual discrimination experiment to 11 subjects (9 male, 2 female). Each of the subjects has completed the following experiment, consisting of several levels. Initially, the subjects were instructed about the experiment. At the beginning of each level, two different tissue stress/strain curves were selected as stimuli: one (labeled as f S ) representing softer tissue, and the other (called f R ) for more rigid tissue. The subjects were allowed to probe the two simulated tissues until they felt comfortable with recognizing the stimuli. Subsequently, each subject performed 16 discrimination tests. The stimulus was chosen at random by the program. The subjects were allowed to interact with the tissue for 7.5 s and were then asked to tell the tissue type they had felt. The answer was typed into the program, which then revealed whether the guess was right or wrong. Neither the operator knew the correct answer beforehand. After each guess, the subject was told the right answer, so to enhance the his/her training. The subject had to provide an answer at each test: when the subject was uncertain, he/she was asked to make a guess. All the answers were logged. At the end of the level, the subjects were allowed a short break. At the first level, f S = f v and f R = f w are selected by setting t S = 0 and t R = 1, respectively. At the following levels, the experiment was conducted in the same way, but the discrimination was made more difficult by choosing closer curves. If we define dt := t R t S, we have dt 0 = 1 for the first level. We diminished dt progressively: dt i+1 = dt i /k, k > 1. By choosing k = , so that the range of the curves shrinks to 10% of the original range in 5 levels, we obtained the curves plotted in Figure 3 that we used for the experiments.

4 Normalized force 0-0,1-0,2-0,3-0,4-0,5-0,6-0,7-0,8-0, ,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 Normalized displacement Figure 3: The interpolated normalized displacement/force curves used in the five levels of the experiments. S0 and R0 are the soft and rigid tissues of level 0. As the level number increases, the curves are more similar. To assess discrimination capability, we tested the subjects responses against the null hypothesis that the subject was guessing the tissue type randomly, not feeling any real difference. If this is the case, we expect the subject to guess correctly 50% of the answers. This behaviour corresponds to a binomial distribution of probability p = 0.5. On the other hand, if the subjects score is above 50%, we need to verify if this difference is statistically significant. We decided a priori to reject the first and last two tests, to reduce the effects due to insufficient training and fatigue, keeping only the 12 central answers. The probability of the binomial distribution of making k or fewer mistakes in n tests by guessing randomly is: ( ) n n p(k ) = i i=n k 1 2 n. In our case n=12, which leads to p(0 ) = , p(1 ) = , p(2 ) = , p(3 ) = If we chose p < 5% as the statistical significance level, we require that the subjects make less than three mistakes to conclude that the two simulated tissues were discriminated successfully. If the subject discriminated successfully the stimuli at one level, he/she accessed the following level, and performed the tissue discrimination experiments with closer curves, as described above. On the other hand, if discrimination was unsuccessful, he/she repeated the level at most once before continuing to the next level. The same experiment was performed with two feedback modes: the first time both haptic and visual feedback was available to the subject; the second time, the subject had to discriminate the tissues only by means of haptic feedback. When only haptic feedback was present, the subjects were S0 R0 S1 R1 S2 R2 S3 R3 S4 R4 not allowed to watch their grasping hand or the haptic interface, nor to close their eyes, so that some sort of visual input was always present. To minimize the learning effect, the experiments were performed in separate sessions by providing combined haptics and visual feedback on the first session, and only haptics feedback on the second session, carried out at least one hour later. To test if the presence of visual feedback affects the discrimination capability, we used Wilcoxon s matched-pairs signed-rank test [Gla01], a non-parametric alternative to the paired t-test, on the number of mistakes made by each subject in the two feedback modes. We performed Wilcoxon s test on the number of mistakes for each level separately, and on different combinations of levels. 3. Results Table 1 summarizes the number of mistakes for all subjects, with only haptic feedback (H), and combined visual and haptic feedback (HV ). We see that all subjects performed equally well on level 0, which means that the stimuli were so different that 100% discrimination could be achieved by haptic feedback alone. At higher levels, the subject made mistakes and their average number increased as the level increased. Also, from Table 1, we see that the number of experiments in which significant discrimination (according to the binomial distributions) was reached only at second attempt, increases as the level increases. Table 2 shows results of Wilcoxon s signed-rank test analysis. p(n,w) is the probability of observing the given result, or one more extreme, by chance if the null hypothesis ( visual feedback is irrelevant ) is true. Small values of p(n,w) cast doubt on the validity of the null hypothesis. Though the number of errors in the absence of visual feedback is clearly higher at levels 2 and 3, the low number of subjects did not allow to reject the null hypothesis at the 5% level (p(n,w) < 0.05). Nevertheless, we can reject the null hypothesis at level 4 and, if we consider the union of several levels, we can see that in all combinations of two or more levels, the null hypothesis can also be rejected at the 5% level. Including level 0 and 1 in the global analysis makes no difference, since the subjects performed almost equally well with both feedback modes, and data pairs for which difference is 0 are excluded from the analysis [Gla01]. In Table 2, n is the number of matched-pairs with non-zero difference. 4. Discussion From the results we can conclude that subject made a higher number of errors as the stiffness difference between stimuli decreased. They therefore also were more often unable to pass the levels of the experiments at the first attempt. For levels 0 and 1, almost 100% accuracy was achieved and the presence of visual feedback was statistically irrelevant. The presence of visual position feedback reduced the

5 Level Feedback HV H HV H HV H HV H HV H A * 2 B * 5* 2* 6* C * * D * 1 3* 5* 9* E * 2.5* 1 5* F * 2 2 4* 1 1 G * 1 H * 1 5.5* 3.5* 6* I * * 6.5* J * 2 4* 3* 6* K * 3.5* 2 5* Average Table 1: Number of mistakes for all subjects, at various levels (numbers on first row) and feedback modes (2nd row: HV means combined haptic and visual feedback, H means only haptics. Numbers followed by an asterisk mean that the test was not passed at the first attempt, since in the number of correct answers was not statistically significant. In this case, we counted the average number of mistakes done in the first and second attempts. The last row reports the average number of mistakes. Level n W p(n,w) h(n,w) All Table 2: Wilcoxon s signed-rank test results at levels 0 4. Subsequent rows have been computed by combining the number of mistakes among different levels, e.g. 2+3 means the union of data sets of level 2 and 3. n is the number of matched-pairs with non-zero difference, W is sum of the signed ranks, p(n,w) the associated probability, and h(n,w) = 1 if the null hypothesis can be rejected at the 5% level (p(n,w) < 0.05), otherwise h(n,w) = 0. number of average mistakes at all higher levels (see Figure 4). The presence of visual feedback reduces the average number of mistakes by 33-40% at levels 1, 3, and 4, and reaches a peak of 59% at level 2. The average reduction considering all 5 levels is 42%. Due to the low number of subjects, in single-level analysis, statistical significance, h(n,w) = 1, was only reached at level 4 of the experiments. Level 2 is close to the 5% level, while level 3 is at 9%. However, the levels were chosen arbitrarily and the aim of the experiment was to assess the influence of visual feedback globally, at all levels. If we join the data from two or more levels, we can reject the null hypothesis at the 1% level: p(n,w) < 1%. If we join the data from all levels, p(n,w) 1%. Average number of mistakes HV H mean 33% 59% 40% 36% Level Figure 4: The average number of mistakes made by subjects at all levels. Average number of mistakes done with different feedback modes are represented in bars. The average number of mistakes, made independently from the feedback type, is represented as continuous plot. Numbers floating over the HV bars show the percentage reduction of errors due to the presence of visual feedback (H HV)/H. If some residual learning effect was present in the second experimental session, this should have allowed subjects to better discriminate the haptic stimuli and to reduce the p(n,w) score. We chose to perform the experiments with only haptic feedback as second, to be conservative on the computation of the statistical significance level. Since the stimuli were chosen at random between the soft and hard tissue, several subjects reported that they felt it

6 more difficult to answer after a series of identical tissues in the same level, rather than when the tissues changed more often during the same level. Some subjects felt that visual feedback was more useful as the level increased, i.e. as the difference between soft and rigid tissue decreased. To support in part this statement, we notice that the score of Wilkinson s test is only significant at level 4. Some subjects reported that they felt that visual feedback distracted them during the tests, especially at higher levels. However, their performance was aligned with those of the other subjects and they too scored worse without visual feedback. The 130 Hz limit on the update rate of the haptics loop did not allow the users to interact too fast with the virtual tissue, e.g. impulse responses were not simulated reliably. However, the update rate was sufficient to simulate typical probing that could be done safely during surgery. We are currently developing a new application with the simulation running at 1 khz in the haptics loop. 5. Conclusions This paper shows the importance of visual position feedback in haptic stiffness discrimination tasks. In a binary soft/hard tissue discrimination experiment, subjects scored significantly better when also visual feedback about the probe position was present, rather than with only haptic force feedback. This finding can be important for the development of sensorized small probes and catheters to be used in minimally invasive surgery procedures. In the future we will further investigate the importance of visual feedback, by performing experiments without haptic feedback, by introducing distractors, by comparing several grasping/probing tasks, and by assessing the effect of a more realistic simulation of the tissue. References [BDK 04] BASDOGAN C., DE S., KIM J., KIM M. M. H., SRINIVASAN M. A.: Haptics in minimally invasive surgical simulation and training. IEEE Computer Graphics and Applications 24, 2 (March/April 2004), [BHM 99] BHOLAT O. S., HALUCK R. S., MURRAY W. B., GORMAN P. J., KRUMMEL T. M.: Tactile feedback is present during minimally invasive surgery. J Am Coll Surg 189, 4 (Oct 1999), [Bur00] BURDEA G. C.: Haptics issues in virtual environments. In Proc Intl Conf Computer Graphics (Washington, DC, USA, 2000), IEEE Computer Society, pp [ÇWTS01] ÇAVUŞOĞLU M. C., WILLIAMS W., TENDICK F., SASTRY S.: Robotics for [DCM 00] telesurgery: second generation Berkeley/UCSF laparoscopictelesurgical workstation and looking towards the future applications. In Proc.the 39th Allerton Conf. Communication, Control and Computing (2001). DARIO P., CARROZZA M. C., MARCACCI M., ANDB. MAGNAMI S. D., TONET O., MEGALI G.: A novel mechatronic tool for computer-assisted arthroscopy. IEEE Trans Inf Technol Biomed 4, 1 (Mar 2000), [ETM eda] EISINBERG A., TONET O., MACRÌ G., CARROZZA M., DARIO P.: Haptic tissue discrimination by means of microfabricated instrument in fetal cardiac surgery. Virtual Reality (2005, submitted). [ETM edb] EISINBERG A., TONET O., MACRÌ G., CARROZZA M. C., DARIO P.: Microfabricated instruments for fetal cardiac surgery: experiments on haptic tissue recognition. In Proc. 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems - IEEE Virtual Reality 2006 (2006, submitted). [Gla01] GLANTZ S. A.: Primer of biostatistics, 5th ed. McGraw-Hill Medical, November ISBN= [GMO02] GEROVICHEV O., MARAYONG P., OKA- MURA A. M.: The effect of visual and haptic feedback on manual and teleoperatedneedle insertion. In Proc. MICCAI - LNCS 2488 (Berlin Heidelberg, Germany, 2002), Dohi T., Kikinis R., (Eds.), MICCAI 2002, Springer Verlag, pp [HRH 04] HAGMANN E., ROUILLER P., HELMER P., GRANGE S., BAUR C.: A haptic guidance tool for CT-directed percutaneous interventions. In Proc. 26th Intl Conf IEEE Engineering in Medicine and Biology Society (San Francisco, CA, USA, September ), pp [KDOY05] KITAGAWA M., DOKKO D., OKAMURA A. M., YUH D.: Effect of sensory substitution on suture-manipulation forces forrobotic surgical systems. J Thorac Cardiovasc Surg 129, 1 (Jan 2005), [KH95] [KLM93] KONTARINIS D., HOWE R.: Tactile display of vibratory information in teleoperation and virtual environments. Presence 4, 4 (1995), KLATZKY R. L., LEDERMAN S. J., MAT-

7 [KOB 02] ULA D. E.: Haptic exploration in the presence of vision. J Exp Psychol Hum Percept Perform 19, 4 (Aug 1993), KITAGAWA M., OKAMURA A. M., BETHEA B. T., GOTT V., BAUMGARTNER W. A.: Analysis of suture manipulation forces for teleoperation with forcefeedback. In Proc. MICCAI - LNCS 2488 (Berlin Heidelberg, 2002), Dohi T., Kikinis R., (Eds.), MICCAI 2002, Springer Verlag, pp [RHMS99] [SD97] ROSEN J., HANNAFORD B., MACFARLANE M., SINANAN M.: Force controlled and teleoperated endoscopic grasper for minimallyinvasive surgery - experimental performance evaluation. IEEE Transactions on Biomedical Engineering 46, 10 (October 1999), SCOTT H. J., DARZI A.: Tactile feedback in laparoscopic colonic surgery. Br J Surg 84, 7 (Jul 1997), [KRS03] [LA81] [LVJ 04] [Mas95] KIM H. K., RATTNER D. W., SRINIVASAN M. A.: The role of simulation fidelity in laparoscopic surgical training. In Proc. MIC- CAI - LNCS 2878 (Berlin Heidelberg, 2003), Ellis R., Peters T., (Eds.), MICCAI 2003, Springer Verlag, pp LEDERMAN S., ABBOTT S.: Texture perception: study of intersensory organization using a discrepancyparadigm, and visual versus tactual psychophysics. J Exp Psychol Hum Percept Perform 7, 4 (1981), LÉCUYER A., VIDAL M., JOLY O., MÉ- GARD C., ALAINBERTHOZ: Can haptic feedback improve the perception of self-motion in virtualreality? In Proc. 12th Intl Symp Haptic Interfaces for Virtual Environment andteleoperator Systems (HAPTICS 04) (2004). MASSIMINO M. J.: Improved force perception through sensory substitution. Control Eng Pract 3, 2 (1995), Control Engineering Practice. [STKT 01] [TLBM96] [TPM05] [WSH02] SZILI-TOROK T., KIMMAN G., THEUNS D., RES J., JORDAENS J. R. R. L. J.: Transseptal left heart catheterisation guided by intracardiac echocardiography. Heart 86, 5 (Nov 2001), E11. TAYLOR R., LAVALLÉE S., BURDEA G., MÖSGES R. (Eds.): Computer-Integrated Surgery: Technology and Clinical Applications. The MIT Press, Cambridge, MA, USA, TAVAKOLI M., PATEL R., MOALLEM M.: Haptic interaction in robot-assisted endoscopic surgery: a sensorized end-effector. Int J Medical Robotics and Computer Assisted Surgery 1, 2 (2005), WAGNER C., STYLOPOULOS N., HOWE R.: The role of force feedback in surgery: analysis of blunt dissection. In 10th Int Symp Haptic Interfaces for Virtual Environments and TeleoperatorSystems (Orlando, March ). [MECD03] MENCIASSI A., EISINBERG A., CARROZZA M., DARIO P.: Force sensing microinstrument for measuring tissue properties and pulse in microsurgery. IEEE/ASME Transactions on Mechatronics 8, 1 (March 2003), [MNK 05] MAYER H., NÁGY I., KNOLL A., SCHIRM- BECK E. U., R.BAUERNSCHMITT: An experimental system for robotic heart surgery. In IEEE 18th International Symposium on Computer-Based Medical Systems(CBMS) (Dublin, Ireland, June 2005). [NMK04] NÁGY I., MAYER H., KNOLL A.: Application of force feedback in robot assisted minimally invasivesurgery. In Eurohaptics (Munich, Germany, June 2004). [PBF97] PLINKERT P. K., BAUMANN I., FLEMMING E.: Tactile sensor for tissue differentiation in minimally invasive ENT surgery. Laryngorhinootologie 76, 9 (Sep 1997),