Face, content and construct validation of the first virtual reality laparoscopic nephrectomy simulator

Transcription

1 29 THE AUTHORS. JOURNAL COMPILATION 29 BJU INTERNATIONAL Original Articles BREWIN ET AL. BJUI BJU INTERNATIONAL Face, content and construct validation of the first virtual reality laparoscopic nephrectomy simulator James Brewin, Tim Nedas, Ben Challacombe, Oussama Elhage, Jonas Keisu* and Prokar Dasgupta Urology Centre, Guy s Hospital and MRC Centre for Transplantation, NIHR Biomedical Research Centre, King s College London, London, UK, and *Mentice, Gothenburg, Sweden Accepted for publication 22 October 29 Study Type Therapy (case series) Level of Evidence 4 OBJECTIVE To evaluate the face, content and construct validity, and to identify whether participants improved with practice, for the Procedicus MIST Nephrectomy TM simulator (Mentice, Gothenburg, Sweden), which incorporates force feedback and can record numerous performance measures (metrics) during the simulation, and which is the first virtual reality simulator for laparoscopic nephrectomy. SUBJECTS AND METHODS Eight expert urological laparoscopic surgeons, 1 novices and 1 trainee urologists completed at least one simulated retroperitoneal radical nephrectomy. They completed a structured questionnaire to assess face and content validity; the performance of experts and novices were then compared to evaluate construct validity. RESULTS Face validity was established by the experts who all felt the simulator was a good training tool. Content validity was established by the experts who rated all aspects of the simulator as above average for realism. When performance metrics were analysed, experts completed the simulated nephrectomy significantly faster than novices, with fewer errors, less simulated haemorrhage and less tool travel, which established construct validity. After practice, both novices and trainees were able to perform the simulation faster, with fewer errors, less blood loss and less tool travel. CONCLUSIONS This study has established the face, content and construct validity for the Procedicus MIST Nephrectomy. The simulator can distinguish experts from novices and it has potential as a laparoscopic training tool for urology. KEYWORDS virtual reality, laparoscopy, nephrectomy, validation, simulation INTRODUCTION The traditional Halstead apprenticeship model of surgical training has been widely used across Europe and North America, but in recent years this concept has shifted towards one of proficiency-based training [1]. It is no longer acceptable to attempt a surgical procedure without appropriate training and there is a growing need to move the learning curve out of the operating room [2,3]. Along with increasing demands for patient safety there is growing pressure on the training time available for junior doctors, as a result of the 1993 European Working Time Directive that restricts trainee working to 48 h/week from August 29. Furthermore, surgical practice has developed rapidly over the last two decades, particularly in the area of minimal access surgery [4], thus the trainee urologist is faced with a shortened training time and the need to achieve competency in an ever-expanding, and increasingly complicated, number of operations. Simulation has emerged as a tool that, if appropriately integrated into surgical training, might provide a time-efficient, costeffective and safe method of training [2]. Consequently there is a growing consensus of opinion amongst surgeons and urologists that simulation should be a compulsory part of surgical training [5,6]. There are numerous surgical simulators available which including mechanical trainers (e.g. box trainers), animal laboratories and virtual reality (VR) simulators. VR simulators have the advantage that they can monitor user performance and provide feedback to guide training and develop proficiency-based surgical curricula. Furthermore, there is evidence from several randomized trials that VR surgical simulation can improve operating room performance [7,8]. Laparoscopic surgery is a particularly difficult skill to learn and requires different skills from those used in open surgery [9]. The trainee surgeon must learn to operate in a three-dimensional environment using a two-dimensional monitor and tactile (haptic) feedback from the 85 JOURNAL COMPILATION 21 BJU INTERNATIONAL 16, doi:1.1111/j x x

2 FIG. 1. The Procedicus MIST Nephrectomy VR simulator. Expert Age Sex n LRNs n Laparoscopic procedures 1 56 M >2 > M 7 >1 3 4 M >1 > M >5 > 5 42 M >5 > 6 36 M M M 52 1 TABLE 1 The demographics and experience of the eight experts FIG. 2. Dissection of the renal hilum: The renal artery can be seen with a few strands of fibrous adventitia. These need to be dissected off the artery before it can be clipped and cut. Note the bleeding from the hilum. Kidney within Gerotas fascia Renal artery Adventitia instruments as their guide. Furthermore, the surgeon must learn to compensate for the fulcrum effect and use laparoscopic instruments which only have limited degrees of freedom of movement. Several VR laparoscopic simulators have been developed, but until recently there were none specifically designed for urology. For these reasons the Procedicus MIST Nephrectomy TM was developed as a joint venture between Guy s and St Thomas, and Mentice AB, a medical simulation company based in Gothenburg (Sweden). The prototype launched in December 27 simulates both transperitoneal and retroperitoneal laparoscopic radical nephrectomy (LRN) (Fig. 1). Before a surgical simulator can be integrated into the surgical curricula it must be shown to be realistic and teach the skills needed in the operating room [1,11]. Validity is defined as the property of being true, correct and in conformity with reality and it is considered an essential part of simulator assessment [12,13]. Face validity and content validity are both subjective assessments of a test or a training tool by experts. Face validity is established by experts who evaluate a simulator and decide if it tests or teaches what it was designed to test or teach. Content validity is based on a detailed examination of the individual components of a simulator [12]. Construct validity is evaluated by assessing the ability of the simulator to differentiate experienced from inexperienced surgeons. Before a simulator can be used to assess competence it must be shown that simulator performance correlates with user performance, and construct validity is part of this assessment [1]. The aim of the present study was to evaluate face, content and construct validity of the Procedicus MIST Nephrectomy, to establish an evidence base for the use of this new simulator in urology. SUBJECTS AND METHODS The Procedicus MIST Nephrectomy uses a Dell Intel Pentium dual core computer which has USB connections to a keyboard, mouse, three foot pedals, haptic devices (with instruments) and two monitors, one of which is a touch screen. All of the metrics are recorded by the simulator in real time using similar programming to the MIST-VR trainer. The simulator has full haptic feedback provided by Xitact TM -Instrument Haptic Port devices, so that the virtual tissues can be felt by the user during the simulation. The LRN simulation is divided into three tasks. The first is dissection and division of the ureter. The user is presented with a view of the retroperitoneum after balloon dissection of the retroperitoneal space and must start the simulation by identifying the gonadal vein and the ureter. The user must then dissect the ureter from its adventitia and, when free from the adventitia, it must be clipped and divided. For the second task the hilar fat must be dissected to identify the renal artery and vein. The fat is capable of bleeding as in real life (Fig. 2). When the renal vessels are identified they must be cleared of their adventitia then clipped and divided. The final task is to dissect the kidney completely Eight expert laparoscopic urological surgeons, 1 urology trainees and 1 novices volunteered for and participated in the study. All experts had performed over 35 LRNs (range 36 5). Seven were consultant laparoscopic urologists and one was a senior laparoscopic fellow; six of the experts are tutors for laparoscopic skills courses. The demographics and laparoscopic experience of the experts are summarized in Table 1. The urology trainees were a heterogeneous group ranging from one trainee who had performed eight LRNs under supervision to another who had only assisted in 1 laparoscopic procedures. The novice group was composed of eight medical students and two work-experience students who had all observed but not participated in a LRN. All participants performed at least one retroperitoneal LRN on the Procedicus VR simulator, and each of the three tasks were completed in order. All 1 of the novice participants and seven of the 1 intermediates repeated the simulation. Participants were orientated to the simulator and supervised by the same investigator. All participants watched demonstration videos and read the instructions embedded in the simulator before starting each task. After the simulation the experts completed a structured JOURNAL COMPILATION 21 BJU INTERNATIONAL 851

3 BREWIN ET AL. FIG. 3. Assessment of content validity by the eight experts. Task 1, dissection and division of the ureter; Task 2, dissection of hilum and division of hilar vessels; Task 3, dissection of kidney from peritoneum; Likert score: 5, very realistic; 4, realistic; 3, average; 2, poor; 1, very poor. Mean Leikart score TABLE 2 Analysis of expert and novice performance, the performance of novices during their first and second simulations, and the performance of seven trainees during their first and second simulations Median (interquartile range) Group Task time, s Haemorrhage, ml Tool travel, m Total errors, n Novices (1) 224 ( ) 111 ( ) 37.9 ( ) 419 (296 81) Experts (8) 131 ( ) 236 ( ) 24.5 ( ) 181 ( ) P* Novice initial (1) 224 ( ) 111 (585 16) 37.9 ( ) 419 (296 81) Novice repeat (1) 136 ( ) 544 ( ) 29.3 ( ) 337 (23 625) P Trainee initial (7) 143 ( ) 334 ( ) 25.3 ( ) 229 (172 37) Trainee repeat (7) 981 ( ) 23 ( ) 21.5 ( ) 195 ( ) P Graphics Instruments Haptics Vessles Ureter Fat Kidney Task 1 Task 2 Task 3 *Mann Whitney U-test; Wilcoxon matched pairs. questionnaire to evaluate face and content validity, and the trainees completed a short questionnaire asking if they thought that the simulator was a valuable training tool. The performance metrics of the novice and expert groups were compared to assess construct validity. As the data were not normally distributed the nonparametric Mann Whitney U and the Wilcoxon matchedpairs tests were used as appropriate, with a two-tailed significance level of P <.5 considered to indicate significance. The Jonckheere-Terpstra (JT) test, a nonparametric ANOVA for ranked groups, was used to assess whether there were statistically significant differences between the trainee, novice and expert groups, with a two-tailed significance level of P <.5 considered to indicate significance. RESULTS FIG. 4. Comparison of novice, trainee and expert performance (median with interquartile range). The differences between groups were all statistically significant (P <.1, JT test). Task time, sec Errors Task Time Errors Haemorrhage, mls Tool travel, m Haemorrhage Tool Travel All eight experts thought that the simulator was a good training tool for LRN. Six of the eight experts felt the simulator would be a good training tool for laparoscopic surgery in general. This opinion was supported by the intermediate group; all 1 intermediates thought that the simulator was a good training tool for both LRN and for laparoscopic training in general. The experts also completed a structured questionnaire and graded the realism of specific components of the simulator using a five point Likert scale with one being the lowest (very poor) and five being the highest (very realistic). As shown in Fig. 3, all components of the simulation scored a mean of 3 (average realism). The graphics and instrument movements were felt to be particularly realistic (mean 3.9 and 3.8, respectively) and the most realistic task was dissection of the kidney from the peritoneum (mean 3.9). Construct validity was assessed by comparing the performance of the expert and novice groups. The experts completed the simulation significantly faster than novices, with less tool travel, less simulated haemorrhage and fewer errors (P <.5, Mann Whitney U-test; Table 2). These statistically significant differences between the performance of experts and novices therefore established construct validity for the metrics of task time, haemorrhage, tool travel and errors. The trainee group generally performed better than novices but worse than experts, as shown in Fig. 4. The median performance scores of the groups were statistically significant (by the JT test) for haemorrhage (experts 236, vs trainees 377, vs novices 852 JOURNAL COMPILATION 21 BJU INTERNATIONAL

4 111 ml, P <.1), errors (181 vs 294 vs 419, P <.1), task time (131 vs 1459 vs 224 min, P <.1) and tool travel (24.5 vs 28.4 vs 37.9, P <.1). This further supports construct validity, as the simulator can differentiate novices, trainees and experts. All 1 of the novice participants and seven of the 1 trainees repeated the simulation. When novices repeated the simulation the median task time, tool travel and errors all significantly improved (P <.5, Wilcoxon matched-pairs test); novices also performed the second procedure with less blood loss but this was not statistically significant (Table 2). The trainees performed the second simulation significantly quicker and with less blood loss than the first simulation (P <.5, Wilcoxon matched-pairs test) but, although the other metrics of performance improved, they were not statistically significant (Table 2). DISCUSSION Several VR laparoscopic simulators have been shown to improve performance in the operating room, but the Procedicus MIST Nephrectomy is the first urology-specific laparoscopic simulator. Face and content validity were established by eight experts who all felt that the simulator was a good training tool for LRN, and this was supported by the opinion of the trainees. Content validity was established by expert assessment of the simulator, and all aspects of the simulator scored at least average for realism. A simulator must be realistic enough to teach the skills required in real life but it probably does not have to be an exact replica of real surgery to achieve this. Although the simulator was not rated as very realistic by the experts (Fig. 3) all of the experts thought it was a good training tool. It therefore seems likely that the simulator is realistic enough to teach the skills required to make it an effective laparoscopic training tool. These face and content validity results support the use of the simulator as a training tool but they are subjective measures of validity. Construct validity is an objective measure of validity; by comparing the performances of experts and novices, we found that experts outperformed novices during LRN. This is a well-established method for evaluating construct validity and numerous authors have established construct validity for surgical simulators in this manner [14 16]. We also showed a difference in performance between the trainee, expert and novice groups which further supports construct validity. Most authors feel that construct validity is the minimum amount of evidence needed to use a simulator as an assessment tool [1]. As we have established construct validity it is reasonable to use the performance metrics to provide constructive feedback to help trainees improve their performance during simulation training. However, more research would be needed to establish predictive validity, reliability and feasibility before these metrics could be used as formal assessment tools during training. Our study correlates with several other validation studies of laparoscopic simulators, which have found that task time and errors distinguish experts from novices [16 19]. Other authors have also found that experienced surgeons perform laparoscopic tasks more smoothly and with less instrument movement than novice surgeons [2,21]. Although the Procedicus MIST Nephrectomy does not measure smoothness of movement, our study supports the finding that total instrument movement (tool travel) can distinguish experts from novices. Haemorrhage has been shown to correlate with performance during simulated TURP [22] but this is the first study to show that simulated haemorrhage during VR laparoscopy can distinguish experts from novices. It is known that experienced surgeons perform laparoscopic tasks in the operating room faster, with fewer errors and better economy of movement than less experienced surgeons [2,21,23]. That experts outperformed novices on the simulator for these metrics suggests that the simulator is a reasonable approximation of real surgery, as the experts could transfer their laparoscopic skills from the operating room to the simulated environment. It has been shown that novices and trainees improved with practice on the simulator, and as the simulator approximates real surgery these skills might be transferable to the operating room. However, these improvements in performance might represent participants learning the simulation rather than developing real transferable laparoscopic skills. A randomized controlled trial would be needed to compare the operating room performance of a simulatortrained group to a group without training to prove that the skills learnt on a simulator are transferable to the operating room. In conclusion, this study has established the face, content and construct validity for the Procedicus MIST Nephrectomy, a novel VR simulator that simulates LRN. The metrics of task time, tool travel, haemorrhage and errors can distinguish experts from novices and it is reasonable to use them to provide informal feedback to aid learning. The simulator appears to be a good training tool for LRN. ACKNOWLEDGEMENTS This work was kindly supported by a project grant from the Guy s and St Thomas Charity. We also acknowledge financial support from the Department of Health via the National Institute for Health Research (NIHR) comprehensive Biomedical Research Centre award to Guy s & St Thomas NHS Foundation Trust in partnership with King s College London and King s College Hospital NHS Foundation Trust. We are grateful to all experts, trainees and novices for sparing their valuable time and to Nigel Smeeton for his assistance with the statistics. The development and validation of this simulator has been awarded two consecutive Chisholm Gold Medals by the University of London. CONFLICT OF INTEREST Jonas Keisu is employed by Mentice; although he was not involved in the validation studies he was instrumental in simulator development and continues to provide technical assistance. REFERENCES 1 Satava RM. The future of surgical simulation and surgical robotics. Bull Am Coll Surg 27; 92: Aggarwal R, Darzi A. From scalpel to simulator: a surgical journey. Surgery 29; 145: Scott DJ, Dunnington GL. The new ACS/ APDS Skills Curriculum: moving the learning curve out of the operating room. J Gastrointest Surg 28; 12: JOURNAL COMPILATION 21 BJU INTERNATIONAL 853

5 BREWIN ET AL. 4 Darzi A. Saws and Scalpels to Lasers and Robots Advances in Surgery. London: Department of Health, 27 5 Le CQ, Lightner DJ, VanderLei L, Segura JW, Gettman MT. The current role of medical simulation in American urological residency training programs: an assessment by program directors. J Urol 27; 177: Korndorffer JR Jr, Stefanidis D, Scott DJ. Laparoscopic skills laboratories: current assessment and a call for resident training standards. Am J Surg 26; 191: Seymour NE. VR to OR: a review of the evidence that virtual reality simulation improves operating room performance. World J Surg 28; 32: Aggarwal R, Ward J, Balasundaram I, Sains P, Athanasiou T, Darzi A. Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Ann Surg 27; 246: Figert PL, Park AE, Witzke DB, Schwartz RW. Transfer of training in acquiring laparoscopic skills. J Am Coll Surg 21; 193: McDougall EM. Validation of surgical simulators. J Endourol 27; 21: Aucar JA, Groch NR, Troxel SA, Eubanks SW. A review of surgical simulation with attention to validation methodology. Surg Laparosc Endosc Percutan Tech 25; 15: Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg Endosc 23; 17: Carter FJ, Schijven MP, Aggarwal R et al. Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc 25; 19: Zhang A, Hunerbein M, Dai Y, Schlag PM, Beller S. Construct validity testing of a laparoscopic surgery simulator (Lap Mentor): evaluation of surgical skill with a virtual laparoscopic training simulator. Surg Endosc 28; 22: van Dongen KW, Tournoij E, van der Zee DC, Schijven MP, Broeders IA. Construct validity of the LapSim: can the LapSim virtual reality simulator distinguish between novices and experts? Surg Endosc 27; 21: Van Sickle KR, McClusky DA, Gallagher AG, Smith CD. Construct validation of the ProMIS simulator using a novel laparoscopic suturing task. Surg Endosc 25; 19: Duffy AJ, Hogle NJ, McCarthy H et al. Construct validity for the LAPSIM laparoscopic surgical simulator. Surg Endosc 25; 19: Seymour NE, Gallagher AG, Roman SA et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 22; 236: Wilhelm DM, Ogan K, Roehrborn CG, Cadeddu JA, Pearle MS. Assessment of basic endoscopic performance using a virtual reality simulator. J Am Coll Surg 22; 195: Darzi A, Smith S, Taffinder N. Assessing operative skill. Needs to become more objective. BMJ 1999; 318: Dosis A, Aggarwal R, Bello F et al. Synchronized video and motion analysis for the assessment of procedures in the operating theatre. Arch Surg 25; 14: Sweet R, Kowalewski T, Oppenheimer P, Weghorst S, Satava R. Face, content and construct validity of the University of Washington virtual reality transurethral prostate resection trainer. J Urol 24; 172: Vassiliou MC, Feldman LS, Andrew CG et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 25; 19: Correspondence: Prokar Dasgupta, 1st Floor, Southwark Wing, Guy s Hospital, Great Maze Pond, London SE1 9RT, UK. prokarurol@gmail.com Abbreviations: VR, virtual reality; LRN, laparoscopic radical nephrectomy; JT, Jonckheere-Terpstra. 854 JOURNAL COMPILATION 21 BJU INTERNATIONAL