Hybrid Visual Servoing for Autonomous Robotic Laser Tattoo Removal IROS2019国际学术会议论文集 1208

Hybrid Visual Servoing for Autonomous Robotic Laser Tattoo Removal Veronica Penza1 Damiano Salerno1 2 Alperen Acemoglu1 Jes us Ortiz1 Leonardo S Mattos1 Abstract Laser tattoo removal is a standard non invasive method for removing color pigments on the skin Increasing number of tattooed people who want to remove their tattoo has driven the medical laser market to develop new technologies for painless scar free and complete tattoo removal However manual use of such laser systems creates post operative com plications since they do not guarantee i protection on non tattooed skin from laser exposure nor ii precise control of the laser focus during the operations for best performance This paper introduces deTattoo a robotic system to improve tattoo removal operations A robotic arm is equipped with a RGB D camera and a visible laser in eye in end confi guration A hybrid visual servoing control is proposed to guarantee the correct pose of the laser with respect to the tattooed tissue while compensating body motions 2D features tracked with a mass spring damper deformable mesh model are combined with the 3D reconstruction retrieved from a RGB D camera in order to build the control law Several experiments were conducted to evaluate the performance of the system with a fi xed or moving tattooed surface at different inclinations Results showed that the proposed framework is able to fulfi l the laser based tattoo removal requirements providing high positioning accuracy 1mm orientation 0 2 and body motion compensation I INTRODUCTION The art of tattoo was already practiced since ancient times in every culture as a way to express thoughts feelings and memories through paintings on the body Its popularity has been continuously increasing in the modern era such that according to a survey conducted in 2015 by The Harris Poll 29 of US adults have at least one tattoo with an grow rate of 8 with respect to the previous 4 years 1 However human nature is inclined to changes one fourth of people with tattoos regret getting them and the increasing demand of tattoo removal creates the need for effective techniques In contrast with different techniques for tattoo removal tested over the years including surgical excision dermabra sion and chemical destruction many of which cause damage to surrounding tissue 2 recent advances have led to the use of laser technologies which has been demonstrated to be more effi cacious and presenting fewer side effects This technology is based on the concept of selective photother molysis a theory described by Anderson and Parrish in 1980 3 Different chromophores melanin pigment water and oxyhemoglobina absorb different wavelengths of laser light 1Veronica Penza Damiano Salerno Alperen Acemoglu Jes us Ortiz and Leonardo S Mattos are with Biomedical Robotics Lab Department of Advanced Robotics Istituto Italiano di Tecnologia via Morego 30 16163 Genoa Italy veronica penza alperen acemoglu jesus ortiz leonardo demattos iit it 2Damiano Salerno is with the Universita degli Studi di Catania Dipar timento di Ingegneria Elettrica Elettronica e Informatica Viale A Doria 6 95125 Catania Italydamiano salerno studium unict it When a chromophore is heated for shorter than its thermal relaxation time i e the time required by the tissue to release 50 of its heat after laser irradiation selective destruction of that chromophore can occur without damaging surrounding tissue Therefore the laser absorption by the tattoo granules strongly affect the effectiveness of the tattoo removal In the current practice an operator has to manually point the laser on the skin surface to remove the tattoo One laser pass should be performed over the desired treatment area with minimal 10 overlap between spots 1 Laser treatments are typically preformed at monthly intervals or longer to permit adequate clearing of ink and appropriate skin healing between treatments Recent studies have shown increased effi cacy of laser tattoo removal when several laser passes are delivered during each treatment session for removal All these aspects make the removal of a tattoo a complicate process which most of the time leave the patient unsatisfi ed The treatment is painful there could be residual tattoo scars and it could take even years to remove a tattoo affecting the patient physically psychologically and socially This paper introduces deTattoo an innovative robotic system to autonomously perform laser tattoo removal by automatizing and optimizing the treatment The system com bines robotics and computer vision in order to i identify and track the unknown tattooed surface area to compensate for body motion ii plan the optimal laser path to avoid tissue thermal damage and iii control a robotic arm to accurately perform the laser positioning and tattoo removal deTattoo is a medical application which has to provide reliability and safety during the tattoo removal process A vision based control strategy is necessary to ensure A constant laser tissue working distance during all the removal process Laser perpendicularity to the surface in order to avoid spot distortion Compensation of body motion during the operations To this end a framework for a hybrid visual servoing approach is presented where the combination of 2D visual features and 3D features obtained from an on line recon structed surface allows the computation of the correct laser trajectory and positioning The fi nal aim of this system would be to achieve the following benefi ts compared to the current procedure a reduced number of treatments due to an improvement in the removal strategy and reduced tissue damage due to a precise and automatic laser path planning The rest of the paper is organized as follows Section II reviews the state of the art on hybrid visual servoing control Section III gives a general overview on the system architecture Section IV presents a 2D deformable mesh 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 IEEE4461 tracking algorithm based on optical fl ow and mass spring damper deformable model for the tattoo motion tracking The strategy to plan the laser path trajectory is described in Section V whereas Section VI describes the proposed hybrid visual servoing control In Section VII we report the experiments conducted to validate the proposed system and results Finally discussion and conclusion are provided in Section VIII II BACKGROUND AND RELATED WORK The aim of vision based control schemes is to defi ne a relationship between the end effector velocity and changes that occur in the observed object in order to minimize an error e t 4 defi ned as e t s s 1 where s is a vector of visual geometrical features identifi ed on the object and s contains their desired values In image based visual servoing IBVS 4 s consists of a set of visual features extracted from the image and consequently its error trajectory is defi ned in the image plane IBVS relies on the online calculation of the image or feature Jacobian and on the distance between the target object and the camera which may be diffi cult to estimate 5 In position based visual servoing PBVS 4 5 s consists of a set of 3D parameters estimated from image measurements and its error trajectory is defi ned in the Cartesian frame PBSV depends on the optical parameters of the visual system and it is very sensitive to calibration errors 5 To overcome these drawbacks hybrid visual servoing approach was introduced in 6 combining a set of 2D and 3D features in s This approach combines object pose estimation and image infor mation improving the robustness of the control Keeping into account the aforementioned considerations about deTattoo application a hybrid approach has been chosen to control the robotic arm based on the information retrieved from the RGB D camera In the context of our application the object shape i e the tattooed surface to scan is unknown and 2D and 3D features must be detected online III SYSTEM ARCHITECTURE deTattoo system is shown in Figure 1 A 6DOF robotic arm is equipped with an RGB D camera and a laser pointer An eye in hand confi guration is used in order to avoid target occlusion during the execution of the trajectory The robotic arm is located above the tattooed surface hereafter for simplicity called object having the tattoo centred in the camera The proposed framework consists in Manual selection on the RGB image of the surface area containing the tattoo and automatic segmentation of the tattoo skeleton Defi nition and initialization of a 2D triangular mesh on the selected area and 2D features detection for tracking purposes Based on the previous steps the laser path along the tattoo skeleton is computed in order to maximise the distance between the delivered laser spots Manual SelectionTattoo Skeleton 2D Mesh and Tracking Path Planning 3D Reconstruction Hybrid Visual Servoing Fig 1 deTattoo system overview For each point representing the tattoo the combination of its position updated at run time based on the mesh tracking and the 3D surface provided by the RGB D camera are exploited for the hybrid visual servoing control IV 2D DEFORMABLE MESH TRACKING The operator s manual selection of the area containing the tattoo allows to isolate the portion of image to be processed and exploited for visual servoing The fi rst step consists in the identifi cation of the tattoo skeleton i e the 2D points pt u v i where u and v are the image pixel coordinates and i the number of points the skeleton is made of These points will be used as 2D features in the hybrid visual servoing approach to position the laser spot for tattoo removal The desired tattoo skeleton is extracted with a simple thresholding technique Due to different factors such as the relative camera object motion or body motion ptineed to be robustly tracked in time At this stage a set of salient features fhis detected We used BRISK feature detector 7 since it was suitable for our application However simple feature based tracking technique would not be suffi cient to update pti since the presence of salient features cannot be ensured along all the tattoo skeleton Motivated by this a geometrical surface model has been used to estimate the motion and deformation of the object providing a deterministic number of control points whose positions is updated based on the tracking of fhfeatures Consequently ptiare linked to the geometrical surface control points and updated based on these links This method can cope with any kind of feature detection and tracking provide it is robust within the specifi c scenario In this work we used a 2D triangular mesh model of N vertices vj ux vy Twith j 1 N and expressed as V v1 vN 2 Any point p within the mesh can be located through a warping function W p V by using the vertices of the trian gle it lies within and its barycentric coordinate bl bm bn where l m n represent the triangle vertex indices and where bl bm bn 1 The warping function is defi ned as W p V b 0 0b v jx vjy T 3 4462 vl vmvn vl vmvn kD k kD cDcD cD kf kf kf vl vmvn kD k k MeshInternal forcesExternal forces fh Fig 2 Mass spring damper deformable mesh model On the left the regular triangular mesh In the center the internal forces edges and angles of a triangle On the right the external forces produced by the features lying inside a triangle The deformation model chosen to update the state vector V and consequently pti is a mass spring damper model 8 as shown in Figure 2 Following this model each vertex of the mesh vjis subjected of internal Fintjand external forces Fextj being the total force Fj Fintj Fextj The internal forces try to keep the shape and dimentions of the triangles while the external forces are produced by the movement of the neighbour features Inspired by 9 the internal forces have two displacement components Fedgeand F The fi rst one is a force between two neighbour vertices viand vjaccording to a spring damper system and computed as Fedgeij Fkij Fcij where Fkij kD vi vj l0 vi vj vi vj 4 Fcij cD vi vj vi vj vi vj vi vj vi vj 5 where kDis the linear stiffness constant cDis the linear damping constant and l0is the initial length of the edge The angular force F produced between two adjacent edges defi ned by the vertices vi vj and vi vk respec tively is calculated as F ijk k 0 where 0is the initial angle k is the angular stiffness constant and vi vj vk arccos v j vi vk vi vj vi vk vi 2 6 Being Ejthe subset of neighbour vertices of vj the internal forces of this vertex are then computed as Fintj X k Ej Fedgejk X i k Ej F ijk 7 The external forces Fextjare calculated as the sum of the forces applied by each neighbour feature A neightbour feature of a vertex vj is defi ned as a feature that lays inside any triangle which contains the vertex vj The force applied by a feature fhis calculated as Ffjh kf fh vj ljh0 fh vj fh vj 8 were kfis the stiffness and ljh0is the initial distance between the vertex vjand the feature fh Consequently begin Hjthe group of neightbour features the external force applied to a vertex vjis calculated as follows Fextj X h Hj Ffjh 9 The estimation of the mesh deformation is computed by solving the dynamic equation of the system taking into account all the forces applied to each vertex using the explicit Euler s method V LASER PATH PLANNING Once the tattoo skeleton points pti are identifi ed and tracked as described in the previous Section IV the se quence in which the laser spots are delivered to these points hereafter called laser path has to be computed in order to maximise the distance between them so as to prevent tissue thermal damage The optimal laser path planning has been addressed as the Travelling Salesman Problem TSP 10 Its most typical representation consists in fi nding given a set of cities and the distances between each pair of them the shortest journey that a salesman must follow to visit all the cities one and only once In our case the cities are represented by the 2D tattoo points ptiand the problem is to fi nd the path with the maximum distance between consecutive points TPS can be modelled as an undirected weighted graph where ptiare the vertices of the graph the paths between them correspond to the edges and a path s distance between ptiand pti 1is the edge s weight di i 1 One of the most used approaches is to fi nd all the possible paths that the laser spot could take in a recursive way The basic idea in recursive TSP approach is to keep track of the nodes that can be navigated starting from each single node and then recursively visit them if they have not already been visited along that path At the end of recursion a tree like structure is created The Held Karp algorithm 11 is used to solve the TPS It proposed the bottom up dynamic programming approach as a solution to improve the brute force method of solving the traveling TPS problem Since the drawback of applying this algorithm in real application is the exponential time complexity O 2nn2 we decided to apply it to subsets of ptiwith i 1 m and m 8 respectively The ptivector is randomly re ordered before applying the algorithm in order to create subsets of points in randomized sequence VI HYBRID VISUAL SERVO CONTROL As previously mentioned hybrid visual servoing is based on the defi nitions of both 2D and 3D features according to the information retrieved from the RGB D camera In the context of our application the object i e the tattooed surface to scan is unknown and 2D and 3D features must be detected and updated online 2D features are represented by pti whose position is linked to the mesh vertices V and updated run time based on the tracking model described in Section IV 3D features are extracted from the depth images 4463 W c c o Fig 3 Transformation involved in deTattoo system as described in details in the following section For the sake of clarity Figure 3 shows the transformations involved in the system A 3D Features Extraction Considering the 3D reconstruction of the object retrieved from the RGB D camera the following 3D features are extracted Orientation u expressed as axis angle parametrization of the rotationc Rcthat the camera has to achieve to move from the desired camera frame to the current one It can be defi ned as cT c cT o oT c 10 wherec Tois the transformation between the desired camera frame c and the object frame o andcTo is the transformation between the current camera frame c and o i e the current object pose in the camera frame In order to build u only the rotational part of matrices in Eq 10 is taken into account Desired orientation can be selected as c Ro I3 that is the condition for the perpendicularity between camera laser assuming that they are collinear and the object Since oT c is time variant and servo dependent it has to be computed at run time as described in Section VI B Depth defi nes the current depth zcwith respect to the desired depth z c as z log zc z c 11 B Online Object Pose Estimation The object posecTowith respect to the camera frame c is defi ned as follows 1 The origin of the object reference frame o is placed on the 3D point corresponding to the current selected 2D point pt 2 z axis is considered as the normal of the local plane fi tting a subset of 3D points within a window n n around pt 3 y axis is the y component of wRc i e the pose of the camera with respect to the world frame w It is obtained from the robotic arm forward kinematics 4 In order to ensure the frame orthonormality x axis is computed as the cross product between y axis and z axis while y axis is refi ned as the cross product between z axis and x axis These steps are computed at each iteration Plane fi tting is performed using RANdom SAmple Consensus algorithm RANSAC to discard outliers and fi nd the optimal result 12 C Visual Servo Control The resulting vector of features is expressed as s ptx pty ux uy log zc z c 12 where ptx pty is the tattoo 2D point to be removed uxand uyare respectively the