Haptic control in a virtual environment

Haptic control in a virtual environment Gerard de Ruig (0555781) Lourens Visscher (0554498) Lydia van Well (0566644) September 10, 2010 Introduction With modern technological advancements it is entirely possible to do remote surgery via a telepresence. This requires the surgeon to use a haptic interface while getting only visual feedback. This poses some obvious limitations to what a surgeon can do, and also poses questions with regard to optimizing the accuracy when spatial and temporal alignment is suboptimal. In the past it has been shown that people can cope with virtual environments [1]. Accuracy of moving a controller to a certain position hardly diminishes when the environment is virtual. However, this research did not include any limitations on the movement of the virtual controller or attempted to increase accuracy. This is the subject of this paper. The scope of previous research is mostly about whether or not subjects are able to perform simple tasks in a real, haptic environment but with a virtual feedback [1]. It has also been shown that when both a haptic and visual stimulus are presented without noise, the visual stimulus will always be dominant [2]. Figure 1: Telesurgery However, the ability to cope with limitations of this virtual feedback or to improve the performance in some way is not part of that research. This leads to the following question: Can people cope with limitations within a virtual environment with haptic controls? There are a lot of possible limitations to the correlation between haptic control and virtual feedback. To limit the scope of this paper only two were chosen, namely gain and latency. These two factors both come from the underlying application in remote surgery. If gain variation would have a positive effect on precision and/or accuracy, this would be of great use within the field of remote surgery. Latency focuses more on the remote part the Internet connects people from all over the world, but the signal has to travel and is bound to be prone to latency. The question is whether or not this affects the performance and if so, by which factor. The experiment uses a similar setup as the one used by Wang and MacKenzie (2000). While simulating a 3D virtual environment, the subject s task is to move a bar into a virtual keyhole, controlled by a physical haptic controller that they cannot see. Both the gain and latency are varied in order to measure the capacity of the test subjects to cope with these limitations and to see whether or not their performance changes. When varying gain, it is expected that accuracy and precision increase if gain is throttled down (i.e. the virtual bar moves slower than the haptic control), since it would be easier to tweak the angle of the solution than it would be at regular speed. However, the solving speed would go up as the subject has to travel a greater distance with the controller. When gain is reversed, the virtual image moves faster than the actual controller. When varying latency, we expect that subjects 1

are precise while accuracy diminishes. It is also expected that solving time will be similar to the control value after the subjects have adjusted to cope with the latency. Methods Cr t Subjects All subjects were right-handed. Subjects needed to be able to see in stereo, since this is a requirement of the setup used in this experiment. There were 10 subjects in total, two of which did not perform the experiment as they were not able to perceive depth using stereo vision. Mi r r or Cont r ol l e r Ta bl e Task The task each subject has to perform is a simple movement task, where they move a virtual object into a virtual keyhole, using a hapfigure 2: Exerpimental setup tic controller. The shape and physics of this virtual object correspond to those of the controller. However, this changes when gain and 15 1 1 centimeters. An Ascension Technology trakstar 6DOF sensor is fitted in the tip latency are varied over predefined ranges. of the pen to return an accurate 3D position and orientation of the pen to the application. Experimental Setup There are no real constraints as to how subjects hold the controller, as long as they hold it in The experimental setup has been made such their right hand and in a manner that feels natthat subjects have the most natural feel be- ural to them. There is one fixed location where tween the controller and the virtual image. a ghost bar will be displayed after each stimsubjects look down on a mirror that reflects ulus. When the subject moves the controller the images of a 120 hertz monitor suspended such that the virtual image lines up with the above it at a 47 degree angle. The mirror ghost image, the next stimulus is presented. is located approximately 40 centimeters above working height. Experimental Design The subject is wearing shutter glasses that switch alternately between transparent and opaque at a 120 Hz rate. This way, a virtual 3D image can be constructed. This is important because the virtual objects are meant to be in 3D. If the subject would view the objects in 2D, his or her accuracy would be lower because it would be hard for the subject to estimate the relative orientation of the keyhole and the pen. In order to quantify differences between subjects and provide insight into the research question, viable metrics have to be used. The research question is about the final positioning of the controller only, not the motion that people employ to get the controller into its final position. The final position is used, and measured when subjects hit the button. The pothe controller is a pen of approximately sition that is measured is the deviation from 2

measured in milliseconds. Figure 3: Screenshot, with the axes of the target shown. the default position (x-, y- and z-coordinates of the pivot points) in millimeters. The orientation (azimuth and elevation) is measured in degrees. Roll is not recorded, since the controller is symmetrical along its longitudinal axis. To use for analysis, the position data of both target and pen is used to calculate the actual angle between target and rod as well as a good distance measure between rod and target to abstract from factors such as the depth that subjects stuck the rod into the target. The angle α between rod and target is calculated by cos(α) = cos(a R A T ) cos(e R E T ), where A R and A T represent the azimuth of the rod and the target and E R and E T the elevations. This angle will be further referred to as angular error. The distance measure used within this experiment is a two-dimensional one. Figure 3 shows the rod, the target and axes originating in the center of the opening of the target. The distance between the rod and the target is defined as the distance between the center of the opening of the target and the intersection of the rod and the xy-plane depicted in figure 3. Thus, it does not matter how far the rod is inserted into the target, only how well it is centered within the opening. This distance will be further referred to as distance error. Another metric used within this research is the time it takes for subjects to complete each trial. The time starts running when the stimulus is presented and ends when the subject hits the button. This is also the moment when the final position is stored. The completion time is The data is stored in a *.csv document every time each subjects presses the button. The conditions of the stimulus (i.e. position and orientation of the target, gain variation and latency variation) are saved, together with both the virtual as well as the actual position and orientation of the bar and the time it took to solve the stimulus. Gain variation Gain variation refers to the movement and rotational speed of the virtual object with regard to the actual speed of the physical controller. It is represented by a factor that multiplies the normal speed, e.g. a gain of 1 is the speed at which the controller and virtual representation move equally fast. A gain of 2 means that the virtual object moves twice as fast as it would normally, relative to the controller. The opposite is also true a gain of 0.5 means that the virtual object is only moving half as fast as the controller. Within the experiment, gain values of 1, 2 and 0.5 are used. There are two types of gain that are varied within the experiment: movement gain and rotation gain. All combinations should be tested due to the factorial design of the experiment. However, a small error was found in the program used during the experiment, which lead to two variations not being tested: slow movement gain and fast rotation gain never occurred in combination, and vice versa. Delay variation Delay variation refers to the delay between movement of the controller and actually moving the virtual object. The control condition has a latency of 0 ms, which means that there is no extra delay before changes in the input are displayed. Increasing delay will add a time delay to this process (i.e. the normal response time of the system + delay). During the experiment, latency values of 100 ms, 200 ms and 300 ms are used. 3

Figure 4: Completion time against delay Figure 5: Angle against Delay Figure 6: Mean angle against Delay Figure 7: Distance against Delay Procedure First each subject s ability to perceive depth stereoscopically is measured by means of a simple test. If subjects should fail this test they could not participate in the experiment. After passing the stereo test, the subjects put on the shutter glasses and assumed the position in which the experiment takes place. They are given the pen and are asked to match the orientation and position of the physical pen with the orientation and position of the virtual pen they see in the mirror. We then enabled the visual feedback to allow the virtual pen to move with the physical pen under control conditions. These steps were repeated until the subject was content with the movement and orientation of the physical pen relative to the virtual pen. The subject was told that their task was to put the pen into the keyhole and that when they are satisfied with the position and orientation, they should press the button. After each trial is complete, a ghost bar will appear and they should put the pen in the ghost pen to present the next stimulus. Furthermore, they were told that if they notice any effects on the movement of the controller, that this is supposed to happen. In total there were two orientations of the keyhole, three gain variations (including normal gain) for both types of gain and four latency variations (including zero latency). Each trial should be completed three times in order to get more stable data, adding up to 72 trials per subject, since delay and gain are not varied at the same time. With about five minutes of instruction and setup and approximately fif- 4

teen seconds per trial, every experiment takes approximately twenty minutes. The order in which the stimuli take place is important, but is also somewhat randomized inside each phase of the experiment. The first 10 trials serve as a baseline with no variation of gain or latency. The second set of 24 trials varies latency only. The third set of 38 trials varies gain only. When the subject starts the actual experiment the lights in the room are turned off to maximize the visibility of the virtual image and minimize the effect of the subject s surroundings. The experimenter stays at a computer next to the experimental setup to ensure everything goes according to the specified procedure. Results After performing eight experiments and removing any invalid trials, a total of 571 measurements is used for analysis. The analysis of the data can be split into multiple sections, each addressing a specific factor of interest in our design: delay variation and gain variation. The results of the measurements are used as a whole and not on a per-subject basis, since it was not the purpose of this study to look into differences between subjects. Delay variation The delay variation data of each subject was divided into four categories: no latency, 100 ms, 200 ms and 300 ms. As a first step in data analysis, boxplots were used to visualize the data (figure 4, 5 and 6). Figure 4 shows the measured completion time for each trial within each category. It is apparent from this boxplot that the control group might have had worse results with regards to time per trial than the measurements in the 100 ms category. However, a steady increase is clearly visible in the 200 ms and 300 ms categories. To confirm whether or not the effect is actually there, a bivariate correlation was calculated which showed a significant correlation between completion time and delay with R = 0.183 and p = 0.012. Figure 5 shows the measured angle between target and pen for each category. When looking at the boxplots, it looks as if some effect might exist. However, analysis of the results show no significant correlations at all. Figure 7 shows the measured distance measure for each trial within each category. It is apparent that the longer the delay, the bigger the distance. However, when running a correlation analysis a significant correlation of R = 0.142 and p = 0.050 is found. Further looking into the bivariate correlation matrix of all measurements with regards to delay showed that there is a significant correlation between time and distance of R = 0.191 and p = 0.008. Thus, the longer a trial took, the smaller the distance. Gain variation Gain variation comes in two flavors: rotation gain and movement gain. Both have been split up into three categories: slow (0.5 times normal speed), normal and fast (2 times normal speed). The data was first inspected using boxplots, similar to the results of the delay variation. Figure 12 shows that measurements with a slow movement gain as well as measurements with a fast movement gain perform very well. However, these differences were insignificant according to a bivariate correlation analysis. According to the bivariate correlation analysis with regards to the angular error, movement gain and angle are not significantly correlated. The results were not further processed because of this. Figure 13 shows a boxplot of the distance error of each trial in each category of movement gain. It is apparent that the measurements with a slow movement gain had a lower distance error, while measurements with a fast movement gain had a higher distance error. According to the bivariate correlation analysis, distance error and movement gain are significantly cor- 5

Figure 8: Movement gain against distance error Figure 9: Movement gain against distance error Figure 10: Rotation gain against distance error Figure 11: Rotation Gain against distance error related with a correlation of R = 0.337 and p = 0.000. Finally, further analysis of the bivariate correlation matrix for the movement gain trials shows that there are no other significant effects within these results. Looking into the means and standard deviations of rotation gain measurements, it is apparent that these are all quite similar. Furthermore, the differences between categories in angular error are all within the assumed precision range of the sensor, so these differences are negligible. After analyzing the results using bivariate correlations, it is confirmed that there were no significant correlations between rotation gain and completion time, angular error or distance error. However, when varying rotation gain, there is a significant negative correlation between completion time and distance error of R = 0.251 and p = 0.003. The only set of trials that has not been discussed so far are the trials that varied both rotation gain as well as movement gain. The boxplots shown in figures a, b, c and d show that there might exist a correlation between gain and distance error, if both movement gain and rotation gain are varied at the same time. Running a bivariate correlation analysis on the data from these trials confirms that there was a significant correlation between movement gain and distance error of R = 0.231, p = 0.000 and between rotation gain and distance of R = 0.107 and p = 0.010. This analysis also shows that there was a negative, but significant, correlation between completion time and distance error of R = 0.131 and p = 0.002. 6

Figure 12: Completion time vs Movement Gain Figure 13: Distance error vs Movement Gain Discussion and Conclusions The questions we were trying to answer have to do with precision, accuracy and completion time. These terms can be translated into mean (precision), standard deviation (accuracy) and milliseconds (completion time). So the question becomes if any expected effects are backed up by the data. Delay variation When varying delay, one would expect that completion time increases and that accuracy and precision decrease. As has been shown in the previous section, neither delay nor gain have a measurable effect on the precision and accuracy of the angular error. However, delay does have a very clear effect on completion time and the precision and accuracy of the distance error, where they get worse as the delay increases. Statistical analysis showed that there was a statistically significant correlation between completion time and delay. Such a statistical significant correlation was also found between distance error and delay. This conforms with what we expected, even though we expected a measurable difference on the angle as well. We conclude that, when faced with delay, subjects tend to take more time to solve each trial correctly, rather than to rush through each stimulus at the same rate they would without any delay. This is backed up by the negative correlation between time and distance error in these trials, which means the larger the completion time of each trial, the lower the distance error. To conclude with regards to delay, we would state that the most prominent factor that increases when delay is introduced, is the completion time of the problem at hand. Subjects seemed to be mostly focused on getting the angle correct at the expense of an increasing distance error as delay increases. This last effect could be caused by the fact that most of the subjects had issues with keeping the pen at a steady position. Without any extra delay this can be compensated. This compensation is increasingly difficult with a delay present, however. Therefore surgeons with steady hands could be much better at it than untrained subjects. Gain variation When lowering gain, one would expect that completion time would increase (since the path or rotation towards the target have been doubled in length) while accuracy and precision increase as well. When increasing gain, the expectations are the direct opposite. The completion time however was not correlated with movement gain nor rotation gain at all. However, with no change to rotation gain, movement gain has a strong correlation with distance error. On the other hand, this was the only case in which completion time and distance error were not correlated. Secondly, looking into rotation gain, there were no significant correlations between rotation gain and completion time, distance error or angular error at all. 7

In this case, though, time does have a negative correlation with distance error. We conclude that varying rotation gain alone has no real impact on precision nor accuracy. When varying both rotation gain and movement gain, the results were much clearer. Both have a positive correlation with distance error, while completion time and distance error had a negative correlation. We conclude that varying movement gain and rotation gain independently has little or no significant effect on performance or completion time. However, varying both at the same time shows that there is a significant effect between gain and distance error - as gain increases, the distance error increases as well. This latter effect was expected and confirmed by the data. On the other hand, when subjects take longer to complete a trial, the distance error is smaller. However, gain does not effect completion time at all by itself, which was not expected. We would like to contribute this to a learning effect. The control values were measured at the beginning of the experiment, while the gain variation took place at the end of the experiment. Therefore, subjects were more adapt at handling the pen than they were at the beginning of the experiment, thus leading to better results in general. Conclusions Some of our expectations, but not all of them, have been met. Delay has a negative effect on completion time of each trial as well as on the distance error. We assume that each trial was harder to complete with higher levels of latency and that compensating for involuntary hand movement is a lot harder with increasingly more latency. We expected to find effects with regards to the angular error, but we could not find one. We conclude that delay and gain have no significant effect on the angular error at all. We assume that subjects focused most on getting the angle correct, at the expense of completion time or distance error. As a side effect, we also found that time has a negative correlation with distance. If a trial took longer, the distance error was generally smaller. We conclude that the best results would be possible if latency would be minimized and both rotation gain as well as movement gain would be lower than normal. If there is no time pressure at all, we assume that better results may be obtained, since the distance error decreases when completion time increases. We do believe that it is possible for surgeons to cope with small amounts of latency if they would ever perform this experiment, due to the fact that their hands are a lot steadier than those of the average subject. Furthermore, it would be interesting to see if the effects of gain and delay combined would either cancel each other out (at low gains) or amplify each other (at high gains). References [1] Yanqing Wang and Christine L. MacKenzie, The Role of Contextual Haptic and Visual Constraints on Object Manipulation in Virtual Environments (2000), CHI Letters 2(1), 532-539 [2] Marc O. Ernst and Martin S. Banks, Humans integrate visual and haptic information in a statistically optimal fashion (2002), Nature 415, 429-433 Increasing gain will also increase the distance error, while having no effect on completion time. We assume that by the time these trials took place, subjects were more adapt at handling the pen than they were when the control measurements were performed. Secondly, we found that movement gain has a larger effect on performance than rotation gain, yet combined they showed an additive effect. 8