Tactile Interactions During Robot Assisted Surgical Interventions Lakmal Seneviratne Professor of Mechatronics Kings College London Professor of Mechanical Eng. Khalifa Univeristy, Abu Dhabi. 1
Overview 1. Surgical Robotics Soft Robots 2. Tactile Interactions - Learning 3. MRI Compatible Force Sensing 4. Haptic Interfaces 5. In Hand Manipulations 2
King s College London 3
Khalifa University Established in 2009 Vision - To be a leading international center of higher education and research 4
1. Surgical Robotics Soft Robots 5
Surgical Robotics - Da Vinci Surgical System www.intuitivesurgical.com 6
Surgical Robotics Robotic Catheterisation Hansen Medical: Robot-steered catheterization tool for cardiac ablation procedures. 7
MIS Tools enter the body through narrow openings and manipulate soft organs that can move, deform, or change in stiffness. Teleoperated Restricted access (through Trocar ports), minimal haptic feedback, rigid robot tools, confined space, safety-critical, real-time, bio-compatible, sterilized tools, MRI compatible. 8
STIFF-FLOP - Inspiration 10
STIFF-FLOP Biological inspiration taken from octopus Soft, highly redundant manipulation device, Embedded distributed sensing cognitive development and intelligent control Learning and cognitive reasoning - learn from physical interactions with environment, 11
STIFF-FLOP STIFF-Flop Concept 12
Octopus-like robot arm EU Project OCTOPUS 13
2. Tactile Interfaces and Learning 14
Tissue Palpation Ex-Vivo Test Rig for Indentation Ex-vivo indentation tests on bovine liver - Measure Force Vs Displacement characteristics 15
Dual Maxwell Model for Palpation Bovine Liver Static indentation (6mm probe) Talal M. Al-ja'afreh, Yahya H. Zweiri, Lakmal D. Seneviratne, and Kaspar Althoefer, A New Soft Tissue Indentation Model for Estimating Force-Displacement' Characteristics using Circular Indenters. Proc IMechE, 2008. 16
Rolling Palpations Ex-Vivo Test Rig (Rolling Device) Horizontal Motion Fz (Normal Force) Vertical Motion Fx (Drawbar Pull) Force/Torque Sensor Wheel Tissue 6-DOF robotic Manipulator 17
Rolling Palpations Soft Tissue Kidney sample with an embedded nodule Test results on pork kidney embedded with simulated tumor Liu, H, Noonan, D. P., Challacombe, B. J., Dasgupta, P., Seneviratne, L. D., Althoefer*, K, Rolling Mechanical Imaging for Tissue Abnormality Localization During Minimally Invasive Surgery, IEEE Transactions on Biomedical Engineering, 2010. 18
Rolling Imaging Rolling imaging from a silicone phantom embedded with 9 nodules 19
Rolling Images 20
Rolling Indentation of Human Prostates Phantom Omni Rolling Probe 2D 3D 21
FE Model for RI 22
3 nodules, 10 mm diameter - Arruda-Boyce model. FE model parameters from unixial tests. Material μ,shear Modulus (kpa) λ m, Locking stretch Mass Density (kg/m 3 ) Type of mesh Rubber (N1) 73.40 1.01 1000 CPS4R FE results are in good agreement with the corresponding experimental data, RMS range 0.02-0.15 % Silicone (RTV6166 gel) 4.98 1.05 980 CPS4R K. Sangpradit, H. Liu, P. Dasgupta, K. Althoefer, and L. Seneviratne, Finite Element Modelling of Rolling Indentation IEEE Transactions on Biomedical Engineering, 2011. 23 23
3. Force Sensing for Surgical Applications 24
Fibre Optic Uni-Axial Force Sensor P. Puangmali, H. Liu, K. Althoefer, and L. D. Seneviratne, Optical Fibre Sensor for Soft Tissue Investigation during Minimally Invasive Surgery, ICRA 2008. 25
Fibre Optic 3 Axis Force Sensor 3-Axis Force Sensor P Puangmali, H Liu, L D Seneviratne, P Dasgupta, K Althoefer. Miniature 3-Axis Distal Force Sensor for Minimally Invasive Surgical Palpation. IEEE/ASME Transactions on Mechatronics. 2011, 26
Fibre Optic Stiffness Sensor Force sensor The miniaturized sensor (11 mm diameter) consists of a force sensor and four displacement sensors Fibre optic technique is applied, MRI-compatible Displacement sensor Panagiotis Polygerinos, Lakmal D. Seneviratne, and Kaspar Althoefer, Triaxial Catheter-Tip Force Sensor for MRI-Guided Cardiac Procedures, IEEE/ASME Transactions on Mechatronics. 2012 27
Fibre Optic Stiffness Sensor Indentation Depth sensing (different Orientations) Force Sensing (a) (b) Mini FID θ z Calibration results of fibre-optic force sensor (c) Silicone Hongbin Liu, Jichun Li, Xiaojing Song, Lakmal Seneviratne, Kaspar Althoefer. "Rolling Indentation Probe for Tissue Abnormality Identification during Minimally Invasive Surgery", IEEE Transactions Robotics. 2011.. 28
Fibre Optic Force Sensing for Cardiac Catheters Wei Yao, Tobias Schaeffter, L Seneviratne. K Althoefer, MR-compatible Catheter for Cardiac Catheterization, ASME Journal of Medical Devices, 2012 29
3-Axis Catheter Force Sensor 12Fr catheter-tip integrated with tri-axial force sensor. Fibre-optic catheter-tip force sensor. Panagiotis Polygerinos, Asghar Ataollahi, Tobias Schaeffter, Reza Razavi, Lakmal D. Seneviratne, and Kaspar Althoefer. MRI-Compatible Intensity-Modulated Force Sensor for Cardiac Catheterization Procedures. IEEE Transactions on Biomedical Engineering, 58 (3), pp. 721-726. 2011. 30
Airflow Force Sensor Constant pressure on 9mm Sphere. Displacement of the ball indicates a change in stiffness of the surface tissue. 31
Air Flow Tactile Probe Direct stiffness indication Tuneable force range (depend on the inlet air pressure) Friction free rolling over the tissue Array of Tactile Element (sphere diameter 4mm) Sensitivity -0.008, diameter 18mm Indika Wanninayake, Lakmal Seneviratne, Kaspar Althoefer, Novel Indentation Depth Measuring System for Stiffness Characterization in Soft Tissue Palpation. IEEE ICRA 2012 32
Air Flow Tactile Probe 33 33
Airflow Force Sensor Experimental Evaluation The location of the 3 nodules on in the silicone phantom correspond to peaks. 34
Airflow Force Sensor Low friction sensor motion Simple design with potential for miniaturisation Can be built from MR-compatible materials 35
Pseudo-Haptic Feedback Real soft tissue Tissue properties Virtual soft tissue Tissue properties Rolling indentati on probe Haptic feedback RMIS system Force matrices of different indentation depths Robot arm Rolling probe Soft tissue Force sensor Palpation input device 36
Palpation simulation system with touch pad as input device 37
Palpation with Pseudo-Haptic Feedback z Virtual force Cursor v Real position x y Soft tissue Tumor 38
Pseudo-Haptic Feedback (a) cursor speed (b) cursor size Min Li, Lakmal Seneviratne, Kaspar Althoefer. Tissue Stiffness Simulation and Abnormality Localization using Pseudo-Haptic Feedback. IEEE ICRA 2012 39
HANDLE EU FP7 IP UPMC (France), Shadow (UK), UC3M (Spain), FCTUC (Portugal), KCL (UK) ORU (Sweden), UHAM (Germany), CEA (France), IST (Portugal) Project Objectives Characterization of object affordances Learning and imitation of human strategies in handling tasks Improving skills through 'babbling' Autonomous in-hand dextrous manipulation 40
Learning Through Touch Hongbin Liu, Lakmal Seneviratne, Kaspar Althoefer. Real-Time Local Contact Shape and Pose Classification using a Tactile Array Sensor. IEEE ICRA 2012 41
Learning Through Touch Right angle edge e 1 e 2 e 3 λ 1 =121.9 λ 2 =22.5 λ 3 =7.5 vertex (corner) e 2 e 3 e 1 λ 1 =30.1 λ 2 =8.45 λ 3 =6.23 e 3 flat surface e 1 e 2 λ 1 =123.6 λ 2 =48.8 λ 3 =3.7 cylinder (r = 6 mm) e 2 e 3 e 1 λ 1 =50.7 λ 2 =20.6 λ 3 =20.5 e 1 e 3 concave surface (r = 45 mm) e 2 e 3 e 1 λ 1 =123.6 λ 2 =53.3 λ 3 =7.5 square (a = 8 mm) e 2 e 1 λ 1 =35.1 λ 2 =16.7 λ 3 =15.7 sphere (r = 6 mm) e 2 e 3 e 1 λ 1 =28.7 λ 2 =18.1 λ 3 =15.6 ring (r outer = 10 mm, r inner = 4 mm ) e 3 e 2 λ 1 =42.2 λ 2 =36.3 λ 3 =37.7 42
Challenges Human-Robot system Human in loop to deal with uncertainties. Monitoring, error recovery Perception Multi-modal (Tactile, Vision, etc) Grasping and Manipulations Tactile Interactions Control, learn Soft systems 43
Ribeiro 44