IROS2019国际学术会议论文集 2232

Abstract Automation in surgery is becoming more and more active in recent years From the editorial of Science Robotics in 2017 there are six levels of autonomy and the existing tele operation lies in level 0 no autonomy Although full autonomy would be fictional in the current moment task shared autonomy would be highly achievable In endoluminal surgeries surgeons rely on manual operation of flexible instruments for tissue manipulation and dissection The techniques are difficult to learn which makes them a good candidate for automation In this paper we introduce our work on shared autonomy of a flexible manipulator in simulated constrained endoluminal tasks e g tissue dissection task in transanal total mesorectal excision TaTME and endoscopic submucosal dissection ESD operations During the procedure the operator defines the dissection trajectory with a Novint Falcon device by marking via points The dissection trajectory is dynamically planned based on the feedback from stereo endoscope system with consideration of the deformation of material Furthermore trajectory of the flexible manipulator is controlled by an adaptive controller with consideration of the possible collision between instruments and surrounding environment This framework is validated by a mock up test with a 3 DOF flexible manipulator Keywords Medical Robots and Systems Sensor based Control Computer Vision for Automation I INTRODUCTION Stomach and colorectal cancers CRC are among the leading causes of cancer deaths worldwide because of their high incidences and mortalities 1 Traditional treatments of the two cancers are laparoscopic surgery and open surgery For early CRC and stomach cancer they could be treated by natural orifice transluminal endoscopic surgery NOTES such as ESD and TaTME Several surgical robotic systems have already been developed for these two surgeries such as ENDOSAMURAI system Olympus Medical system 2 EndoMaster 3 and a flexible surgical robotic system developed in CUHK 4 However all these systems still need Research supported by the Hong Kong General Research Grant with project No 14212316 Early Career Scheme with project No 24204818 and the SJTU CUHK project Xin MA is with the Chow Yuk Ho Technology Centre for Innovative Medicine the Chinese University of Hong Kong Hong Kong e mail maxin1988maxin Peng Wang is with the Chow Yuk Ho Technology Centre for Innovative Medicine the Chinese University of Hong Kong Hong Kong e mail wangpeng 1st MinXin Ye is with the department of Surgery the Chinese University of Hong Kong Hong Kong e mail mye surgery cuhk edu hk Philip Wai Yan Chiu is with the department of Surgery and Chow Yuk Ho Technology Centre for innovative Medicine the Chinese University of Hong Kong Hong Kong e mail philipchiu surgery cuhk edu hk Zheng Li is with the department of Surgery and Chow Yuk Ho Technology Centre for innovative Medicine the Chinese University of Hong Kong Hong Kong e mail lizheng cuhk edu hk surgeons to operate the instruments step by step Surgeons are facing challenges such as a long learning curve operating difficulties mental and physical fatigues Therefore a more automatic approach is desired Considering various tasks involved in TaTME and ESD full automation of entire surgical procedure is currently unachievable As a step forward automation of individual tasks decomposed from these two surgeries such as the dissection is feasible According to the editorial of Science Robotics in 2017 there are six levels of autonomy 5 The robotic systems for TaTME and ESD still lie in level 0 Yet some other surgical robot systems have already developed from no autonomy level 0 to conditional autonomy level 3 The ACROBOT Active Constraint Robot 6 and RIO Robotic Arm Interactive Orthopedic systems 7 are typical systems in level 1 robot assistance which can be operated with active constraints in a shared control modality for orthopedic surgery The ACROBOT can be applied to constrain surgeon s motion in a predefined region and allows surgeons to cut bones more precisely and safely 6 And the RIO system is a highly back drivable six DOF degree of freedom manipulator When the surgical instrument is moved beyond the desired cutting boundaries the system will push back against the users 7 As for level 2 task autonomy some automatic systems for specific surgical tasks have been established such as the STAR Smart Tissue Autonomous Robot 8 and the Probot a computer integrated prostatectomy system 9 With STAR system automatic suturing can be achieved by using a near infrared fluorescent NIRF imaging system and an autonomous suturing algorithm 8 In addition the Probot system is developed to assist surgeons in resecting prostate tissue based on ultrasound images After the resecting area is confirmed by surgeons automatically resection can be achieved However the deformation of the resecting tissue is not considered in Probot ROBODOC 10 and CyberKnife 11 systems are the typical cases in level 3 conditional autonomy The CyberKnife system is an image guided frameless robotic system for radiosurgery and the radiation trajectory can be generated automatically Recently an autonomous control method of flexible manipulators for complex cardiac ablation tasks is proposed 12 Compared with the surgical tasks in gastrointestinal GI tract the tissue deformation in cardiac ablation could be neglected In this paper we present a workflow of shared autonomy dissection in endoluminal surgery inside the GI tract involving surgeon s control in defining the dissection path and the robot s automatic path planning and following in a dynamic environment A flexible manipulator platform is developed to evaluate the proposed method The main contributions of this paper include 1 a workflow of shared Shared Autonomy of a Flexible Manipulator in Constrained Endoluminal Surgical Tasks Xin MA Peng WANG MinXin YE Philip W Y CHIU and Zheng LI Member IEEE IEEE Robotics and Automation Letters RAL paper presented at the 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 Copyright 2019 IEEE autonomy for endoluminal tissue dissection 2 a dynamic trajectory planning and automatic following method which takes tissue deformation and environmental constraints including soft constraints rigid constraints and external disturbance into consideration The rest of this paper is organized as follows In Section II the framework of the shared autonomy is introduced In Section III the motion trajectory generation method based on a stereo endoscope system is proposed In Section IV an automatic control method for a 3 DOF flexible manipulator is detailed In Section V the experimental results are shown At last Section VI concludes this work II FRAMEWORK OF THE SHARED AUTONOMY A Potential Automatic Tasks in Surgery In this paper we consider TaTME and ESD as potential procedures for task shared autonomy As shown in Fig 1 a the procedures of TaTME can be described briefly as follows 13 Firstly the part of rectum 1cm away from the tumor is sutured and the rectal mucosa is circumferentially marked around suture closure Then a circumferential full thickness tractotomy is performed oriented by the marking points The posterior and anterior mesorectal excision dissection is attempted sequentially Finally the tumor is removed and the open distal rectal sump is anastomosed a TaTME process b ESD process Figure 1 The process of TATME and ESD The detailed procedures of ESD are shown in Fig 1 b 14 In brief a circular area 20 30mm containing the lesion is marked by using a cauterizer and then submucosal injection is used to lift the lesion in order to clearly visualize the target area A circumferential incision into the submucosa is then performed around the lesion by using an electrocautery knife Finally the lesion is dissected from underlying deep layers of the GI tract wall and then removed We can see that TaTME is a complex surgery procedure including different surgical tasks such as suturing full thickness rectotomy and dissection Just like TaTME ESD is a time consuming procedure covering tasks such as injection submucosa incision and dissection Therefore full automation of the two surgeries is difficult However both operations involve circumferential target area dissection In the task the surgeon first marks the area to be dissected and then uses the cutting tool to follow the marked trajectory to complete the tissue dissection During the dissection the tissue deforms and the surgeon needs to dynamically change the path of dissection to make sure that only the marked area will be cut off It is considered that this circumferential target area dissection could be automated In conclusion in order to achieve the shared autonomy of a flexible manipulator two technical difficulties need to be overcome the changes of the desired trajectories which are caused by the stretch or deformation of the tissue need to be measured The desired dissection trajectory needs to be re planned accordingly the flexible manipulator may collide with other surgical instruments or organs Therefore an adaptive control method which takes the constraints into consideration is needed 15 16 B Workflow of Shared Autonomy in Tissue Dissection Figure 2 shows the proposed dissection workflow Firstly surgeons need to define the dissection boundary with a flexible manipulator A master device such as the Novint Falcon is used to obtain the 3D information of the operating surgeon s input This 3D information is transferred to a computer for calculating the desired motion of the motors Then a microcontroller is used to receive the commands and implement the motor control The tip position of the flexible manipulator is sensed by an EM sensor to form closed loop control Then a stereo endoscope system is used to monitor the positions of markers defined by the surgeons The 3D information of the markers is used to dynamically plan the tissue dissection path Lastly the flexible manipulator follows the dynamic path with closed loop control automatically Figure 2 The framework of the shared autonomy Marking Injecting Cutting Removing Purse string Marking Tractotomy Mesorectal excision dissection Anastomosing Tumor removed Figure 3 Flowchart of the trajectory generation method III STEREO VISION BASED ONLINE PATH GENERATION In this section a vision based online trajectory generation method is introduced as shown in Fig 3 The 3D information of markers can be reconstructed based on the feature extracted from images firstly Then the desired motion trajectory can be generated by the reconstructed position information of markers A Images Feature Acquisition In order to obtain the 3D information of each green marker around the lesion the 2D coordinates of each green marker on the images need to be extracted firstly The image feature acquisition method is as follows 1 the Gauss filter is used to make image smooth firstly Then green areas on each RGB frame are recognized on the basis of the color based image segmentation method 17 Once the area of interest is detected the RGB frame is converted to gray scale 2 the area based filter method is used to remove the area of interest below a fixed size threshold a a 15 pixels The value of the threshold is determined by the resolution and field of view of the endoscope 3 lastly the center coordinates of all selected area of interest are obtained by using the gray centroid method B B 3D Information Reconstruction The 3D position information of each marker can be reconstructed based on the extracted 2D coordinates As shown in Fig 4 the 3D information reconstruction model can be expressed as 16 0 0 0 0 10000 01000 1 10010010 1 00 00 1 001 0 1 i m i m c i m i m x i m y i m x udxuf y Zvdyvf z x u y v z T T RT 0 RT 0 where x fdx y fdy Rotation matrix R and translation matrix T are the extrinsic parameters of the endoscope f is the focal length of the lens 00 uv represents the image center coordinate and uv represents the extracted coordinate from images c Z is the scale factor dxand dyare the length and width of each pixel on the CCD charge coupled device sensor pi iii mmmm xyz is the measured 3D coordinate of the i th marker The parameters x y 0 u 0 v R T of the endoscope can be precisely calculated based on Zhang s calibration method 18 Figure 4 3D coordinates reconstruction model ll uv and rr uv are the extracted coordinates of markers on the left and right endoscopes respectively C Desired Motion Trajectory Generation Usually the dissection path for TaTME and ESD is a round shape However the path will be elliptical when the target area is stretched by surgical instruments Therefore we assume that the dissection path is an ellipse The cutting trajectory can be expressed as sin cos sin sin cos ee e ee e e e xax yby zbz where ee eee a bxy z are the parameters of the fitted ellipse The least square method is used to fit cutting trajectory based on the reconstructed 3D coordinates of markers pi iii mmmm xyz The object function is shown as follows 222 1 1 sin sin cos iii N emememe eee eee i xxyyzz f a bxy z abb where N is the number of the reconstructed 3D coordinates of the markers It should be noted that the trajectory could be expressed with other curves such as the B spline curve 19 IV AUTONOMUS CONTROL METHOD Figure 5 Structure of the serpentine flexible manipulator Images feature acquisition 3D coordinates reconstruction Motion trajectory fitting Yr Xr Zr Or iiii mmmm pxyz Xl Yl Zl Ol ll uv rr uv Endoscopic visual system Markers A Kinematic Model of the 3 DOF Flexible Manipulator In this paper a tendon driven serpentine flexible manipulator is adopted 20 The structure of the proposed 3 DOF flexible manipulator is shown in Fig 5 H is the length of each vertebra 0 his the length between two adjacent vertebras is the bending angle of each vertebra of the flexible manipulator We assume that the bending angles of each vertebra are the same is the angle between the bending plane and XZ plane t ttt x y z pis the tip position which can be expressed as 21 sin 1 2 cos sin 1 2 sin cos 1 2 tf tf tfb xRN NN yRN NN zRN NNL where 22 1324 2413 2arcsin 2 arctan2 LLLLN d LL LL 0 sin 2 sin 2 f RHhN b L is the movement along the Z axes of the system d is the distance between two wires 0 L is the length of the four wires at initial position of the flexible manipulator And the lengths of the four wires are 1 L 2 L 3 L and 4 L The initial Jacobian can be express as follows J b b b xxyL yyxL zzzL where 2 100 2 200 2 300 2 400 2 cos sin 2 2sin 2 2 sin sin 2 2sin 2 2 cos sin 2 2sin 2 2 sin sin 2 2sin 2 LLN dh LLN dh LLN dh LLN dh B Automatic Dynamic Trajectory Following in Constrained Environment Due to the unpredictable cutting force on the tip of the flexible manipulator and unknown soft and hard obstacles a Jacobian online estimating based automatic dynamic trajectory following method is proposed as shown in Fig 6 Unlike other modeless flexible manipulator control methods 22 23 the initial value of the Jacobian J is calculated from kinematic model of the 3 DOF flexible manipulator system firstly The desired trajectory is dynamically generated with a stereo endoscope system Then the Jacobian can be estimated by using the current position information of the tip cur p obtained from EM sensor Only position information is used in the proposed method The Jacobian online estimation method is shown as follows The manipulator Jacobian J is a configuration dependent function that maps the actuation input velocities l to the end effector velocities t p t pJl Using 7 for each step k the changes in the end effector position for an infinitesimal time period can be estimated based on the changes in the actuation input Jpl k ktk where k l and pk t are the changes in the actuation inputs and the end effector displacements respectively And lk is the pseudo inverse of the lk Then we formulate the following optimization to address this problem l llB pJl k k l kk deskk minimize subject to where des p is the tip position displacement from the current point to the desired goal point k l is the current actuation input at the time step k In this paper 1 1 57 6 28 0 1 m rad radR m B is the joint physical limits We choose k l in order to minimize the motion of each motor and avoid buckling or damaging the manipulator Figure 6 Flowchart of automatic control method of a flexible manipulator In each step k given the actuation control input calculated from 8 the proposed flexible system is moved Then the Jacobian can be estimated by using 9 Here we minimize the J to make transition from the current Jacobian to the next step smoother 1 1 J pJl JJJ kl k tkk kk minimize subject to where k l is the calculated actuation changes from 8 and pk t is the end effector tip position displacement read from an EM sensor A threshold can be specified as the 1 Initial Jacobian J 2 Optimal Control using Jk 3 Estimate Jacobian Jk 1 Endoscopes Desired Trajectory EM sensor Current Position EM sensor Current Position min l lB pJl k k l kk deskk imize subject to l 1 1 min k tkk kk imize subject to J J pJl JJJ pdes pcur Jk 1 J des p termination factor of the algorithm The size of this threshold can be defined experimentally and based on the noise of the sensor feedback In other words the control input and the Jacobian will be updated as long as des p is greater than this threshold value We can see that by using the proposed control method the Jacobian can be estimated even when the flexible manipulator interacting with obstacles or external payloads V EXPERIMENTS A Experimental Platform As shown in Fig 7 the experimental platform consists of three parts a master robot a stereo endoscope system and a 3 DOF wire driven flexible manipulator system Novint Falcon is adopted as the master robot with 1000Hz frequency Its workspace is 10cm 10cm 10cm The stereo endoscope system includes two endoscopes with resolution of 640 480 The frequency of two endoscopes are both 30 fps The