Tactile Feedback of Tool Vibrations! in Robotic Surgery. Katherine J. Kuchenbecker! Haptics Group, GRASP Laboratory! University of Pennsylvania

Transcription

1 Tactile Feedback of Tool Vibrations! in Robotic Surgery Katherine J. Kuchenbecker! Haptics Group, GRASP Laboratory! University of Pennsylvania Workshop at World Haptics June 22, 2015

2

3 Robotic Surgery Image by Intuitive Surgical, Inc.

4 Image by Intuitive Surgical, Inc. Excellent visualization and dexterity......but you cannot feel what the tools are touching.

5

6 Current technology cannot provide perfect haptic feedback in surgical robots like the da Vinci.

7 Are any types of haptic feedback both technically feasible and helpful to surgeons?

8 Using Surgical Instruments Afferent type (and response properties) FA-I (fast-adapting type I) Meissner endings Sensitive to dynamic skin deformation of relatively high frequency (~5 50 Hz) Insensitive to static force Transmit enhanced representations of local spatial discontinuities (e.g., edge contours and Braille-like stimuli) SA-I (slowly-adapting type I) Merkel endings Sensitive to low-frequency dynamic skin deformations (<~5 Hz) Sensitive to static force Transmit enhanced representations of local spatial discontinuities FA-II (fast-adapting type II) Pacini ending Extremely sensitive to mechanical transients and high-frequency vibrations (~ Hz) propagating through tissues Insensitive to static force Respond to distant events acting on hand-held objects Receptive field (and probe) Weak pointed touch Weak pointed touch Light tapping Density (afferents per cm 2 ) SA-II (slowly-adapting type II) Ruffini-like endings Low dynamic sensitivity Sensitive to static force Sense tension in dermal and subcutaneous collagenous fibre strands Can fire in the absence of externally applied stimulation and respond to remotely applied stretching of the skin Data from REFS 6,20. Touch or skin stretch Johansson and Flanagan, Nature Reviews Neuroscience, 2009

9 Tactile! +! contact location! pressure! shear! slip! vibration! temperature Kinesthetic position! orientation! force! torque

10 Tactile! +! contact location! pressure! shear! slip! vibration! temperature Kinesthetic position! orientation! force! torque

11

12 Image by Intuitive Surgical, Inc.

13 sensor Three-axis high-bandwidth MEMS-based accelerometer! $10

14 sensor Works for all tools! No need for sterilization

15 actuator One-axis high-bandwidth voice coil actuator! $300

16 actuator Delivers realistic vibrations! Doesn't interfere with robot

17

18

19 Haptic tool vibration feedback is technically feasible.! Does it help the surgeon? Image by Intuitive Surgical, Inc.

20 Do these vibration signals occur during real surgery?

21 Urologic Procedures on a Porcine Model David I. Lee! Urologic Surgeon Alexei Wedmid! Fellow Examined 30 minutes of surgery. Identified 1404 actions. Karlin Bark, William McMahan, Austin Remington, Jamie Gewirtz, Alexei Wedmid, David I. Lee, and Katherine J. Kuchenbecker. In vivo validation of a system for haptic feedback of tool vibrations in robotic surgery. Surgical Endoscopy, 27(2): , February 2013.

22 Sample Data - Right Tool 82% of actions cause measurable vibrations

23 The intensity of the interaction dictates the strength of the tool s vibration.! 55 of the 1404 tagged events (3.9%) occurred outside the endoscopic camera s field of view or when vision was obscured. Karlin Bark, William McMahan, Austin Remington, Jamie Gewirtz, Alexei Wedmid, David I. Lee, and Katherine J. Kuchenbecker. In vivo validation of a system for haptic feedback of tool vibrations in robotic surgery. Surgical Endoscopy, 27(2): , February 2013.

24 Bariatric Procedures on Human Patients Noel Williams! Bariatric Surgeon Kris Dumon! Bariatric Surgeon Recorded 14 sleeve gastrectomies.! Found significant differences between phases of the operation. William McMahan, Ernest D. Gomez, Liting Chen, Karlin Bark, John C. Nappo, Eza I. Koch, David I. Lee, Kristoffel Dumon, Noel Williams, and Katherine J. Kuchenbecker. A practical system for recording instrument interactions during live robotic surgery. Journal of Robotic Surgery, 7(4): , December 2013.

25 TORS Procedures on Human Patients Gregory S. Weinstein! ENT Surgeon Bert W. O Malley, Jr.! ENT Surgeon Recorded 5 trans-oral tumor resections.! Found significant differences between spatula and grasper vibrations. Ernest D. Gomez, Gregory S. Weinstein, Bert W. O Malley, Jr., William McMahan, Liting Chen, Katherine J. Kuchenbecker. A Practical System For Recording Instrument Contacts and! Collisions During Transoral Robotic Surgery. Presented at Triological Society at COSM, April 2013.

26 Do these vibration signals occur during real surgery?! Yes

27 Do surgeons like haptic tool vibration feedback?

28 2013 SAGES Learning Center in Baltimore

29 haptic and audio feedback of tool vibrations

30 94 Participants Other! 18 Medical Students! 8 Attending Surgeons! 38 Residents! 17 Fellows! 13 Jacqueline K. Koehn and Katherine J. Kuchenbecker, Surgeons and Non-Surgeons Prefer Haptic Feedback of Tool Vibrations in Robotic Surgery. Accepted to Surgical Endoscopy.

31 Do you think it would be useful for you or other surgeons to have the option of using vibrotactile feedback? No! 0% Yes! 100% Jacqueline K. Koehn and Katherine J. Kuchenbecker, Surgeons and Non-Surgeons Prefer Haptic Feedback of Tool Vibrations in Robotic Surgery. Accepted to Surgical Endoscopy.

32 Which type of tool vibration feedback do you prefer? Neither! 2% Only Haptics! 35% Both Haptics and Audio! 62% Only Audio! 1% Jacqueline K. Koehn and Katherine J. Kuchenbecker, Surgeons and Non-Surgeons Prefer Haptic Feedback of Tool Vibrations in Robotic Surgery. Accepted to Surgical Endoscopy.

33 How did tool vibration feedback affect you? 100% 96% 75% 50% 25% 0% Made Me More Aware of Tool Contacts (Audio) 12% Interfered With Ability to Use Robot Jacqueline K. Koehn and Katherine J. Kuchenbecker, Surgeons and Non-Surgeons Prefer Haptic Feedback of Tool Vibrations in Robotic Surgery. Accepted to Surgical Endoscopy.

34 Do surgeons like haptic tool vibration feedback?! Yes

35 How could haptic feedback help the surgeon?

36 TORS Simulator Andrés Bur, M.D.! ENT Resident at Penn

37 A porcine tongue was inserted into the pharynx of an airway mannequin through a hole in the vallecula.! A 5-mm cube of porcine skin was dyed red and inserted in a small pocket in the dorsal tongue to simulate a tumor.! A Crowe-Davis retractor was then used to expose the simulated tumor and tongue base.! A da Vinci Surgical System with a 30 degree endoscope was set up in the usual fashion for TORS tongue base hemiglossectomy.

38 Using the TORS Simulator

39 Haptic Feedback Study 20 otolaryngology residents and 3 fellows.! Randomly assigned to operate with or without haptic tool vibration feedback.! Subject performed 4 simulated TORS posterior hemiglossectomies.! At the start of trial 5, a metal bar was placed to obstruct the spatula cautery, simulating an offcamera collision with the patient s teeth.! Trial was terminated when subject either noticed the obstruction or attempted to begin operating.! Tool vibrations and trial duration were recorded.

40

41 200,000 Subjects who noticed caused lower tool vibrations p = ,469 Total Instrument Vibrations (m²/s²) 150, ,000 50,000 66,219 0 Noticed Collision Did Not Notice Collision

42 100% Subjects with haptic feedback noticed more often p = Percentage Who Noticed Obstruction 75% 50% 25% 76.9% 30% 0% Haptic Feedback No Haptic Feedback

43 Time to Recognize Obstruction (seconds) Subjects with haptic feedback noticed more quickly p = Haptic Feedback No Haptic Feedback

44 How could haptic feedback help the surgeon?! noticing off-camera collisions

45 What do these tool vibrations mean?

46 Is this an experienced robotic surgeon?

47 No. It is a novice. Many awkward motions cause big vibrations.

48 Experienced robotic surgeons tend to do dry-lab tasks faster and with lower tool vibrations than novices. Peg Transfer Needle Pass Suturing Ernest D. Gomez, Karlin Bark, Charlotte Rivera, William McMahan, Austin Remington, David I. Lee, Noel Williams, Kenric Murayama, Kristoffel Dumon, and Katherine J. Kuchenbecker.! Construct validity of instrument vibrations as a measure of robotic surgical skill.! Journal of the American College of Surgeons, 215(3):S , 2012.!! Karlin Bark, Ernest D. Gomez, Charlotte Rivera, William McMahan, Austin Remington, Kenric Murayama, David I. Lee, Kristoffel Dumon, Noel Williams, and Katherine J. Kuchenbecker. Surgical instrument vibrations are a construct-valid measure of technical skill in robotic peg transfer and suturing tasks. In Proc. Hamlyn Symposium on Medical Robotics, pages 50 51, London, England, 2012.

49 What do these tool vibrations mean?! technical skill

50 How would tool vibration feedback affect a surgeon in training?

51 12 general surgery residents! (6 PGY-3 and 6 PGY-7)

52

53 Simulation Setting

54 Gastric Surgery Simulator

55

56 Blinded Video Skill Rating haptic feedback for haptics group no haptic feedback

57 Residents who trained with Blinded Video Skill Rating haptic tool vibration feedback did better surgery. haptic feedback for haptics group no haptic feedback

58 How would tool vibration feedback affect a surgeon in training?! accelerates learning

59 Haptic tool vibration feedback is both technically feasible and helpful to surgeons.

60 K. Kuchenbecker, Ph.D. Raj Aggarwal, M.D.! Engineering Professor GI Surgeon David I. Lee, M.D.! Urologic Surgeon Kris Dumon, M.D.! Bariatric Surgeon Noel Williams, M.D.! Gregory S. Weinstein! Bert W. O Malley, Jr.! Bariatric Surgeon ENT Surgeon ENT Surgeon Ted Gomez, M.D. Will McMahan, Ph.D. Karlin Bark, Ph.D. Liting Chen Michael Gosselin Anna Brzezinski Jamie Gewirtz Jennifer Hui Andrés Bur, M.D. Eza Koch Jackie Kunkel Jackie Koehn Magalie Lilavois Paul Martin Austin Remington Neel Shah Wan Surakiatch Kunal Mahajan John Nappo Charlotte Rivera Dorsey Standish

61 Publications Katherine J. Kuchenbecker, Jamie Gewirtz, William McMahan, Dorsey Standish, Paul Martin, Jonathan Bohren, Pierre J. Mendoza, and David I. Lee. VerroTouch: High-frequency acceleration feedback for telerobotic surgery. In Astrid M. L. Kappers, Jan B. F. van Erp, Wouter M. Bergmann Tiest, and Frans C. T. van der Helm, editors, Haptics: Generating and Perceiving Tangible Sensations, Proc. EuroHaptics, Part I, volume 6191 of Lecture Notes in Computer Science, pages , Amsterdam, Netherlands, July Springer.!! William McMahan, Jamie Gewirtz, Dorsey Standish, Paul Martin, Jacquelyn Kunkel, Magalie Lilavois, Alexei Wedmid, David I. Lee, and Katherine J. Kuchenbecker. Tool contact acceleration feedback for telerobotic surgery. IEEE Transactions on Haptics, 4(3): , July September 2011.!! Ernest D. Gomez, Karlin Bark, Charlotte Rivera, William McMahan, Austin Remington, David I. Lee, Noel Williams, Kenric Murayama, Kristoffel Dumon, and Katherine J. Kuchenbecker. Construct validity of instrument vibrations as a measure of robotic surgical skill. Journal of the American College of Surgeons, 215(3):S , 2012.!! Karlin Bark, Ernest D. Gomez, Charlotte Rivera, William McMahan, Austin Remington, Kenric Murayama, David I. Lee, Kristoffel Dumon, Noel Williams, and Katherine J. Kuchenbecker. Surgical instrument vibrations are a construct-valid measure of technical skill in robotic peg transfer and suturing tasks. In Proc. Hamlyn Symposium on Medical Robotics, pages 50 51, London, England, 2012.!! Karlin Bark, William McMahan, Austin Remington, Jamie Gewirtz, Alexei Wedmid, David I. Lee, and Katherine J. Kuchenbecker. In vivo validation of a system for haptic feedback of tool vibrations in robotic surgery. Surgical Endoscopy, 27(2): , February 2013.!! Jacqueline K. Koehn and Katherine J. Kuchenbecker, Surgeons and Non-Surgeons Prefer Haptic Feedback of Tool Vibrations in Robotic Surgery. Accepted to Surgical Endoscopy, 2014.!! Ernest D. Gomez, Rajesh Aggarwal, William McMahan, Karlin Bark, and Katherine J. Kuchenbecker. Objective assessment of robotic surgical skill using instrument contact vibrations. Accepted to Surgical Endoscopy, 2015.

62 Funding Sources University of Pennsylvania students and colleagues NSF #IIS : CAREER: Haptography: Capturing and Recreating the Rich Feel of Real Surfaces Coulter Translational Research Award: Vibrotactile and Auditory Feedback for Robotic Minimally Invasive Surgery Robot and Tool Donation from Intuitive Surgical

63 Thank you! Katherine J. Kuchenbecker, Ph.D.! Associate Professor, University of Pennsylvania!

64 Questions? Katherine J. Kuchenbecker, Ph.D.! Associate Professor, University of Pennsylvania!