IROS2019国际学术会议论文集1208

Hybrid Visual Servoing for Autonomous Robotic Laser Tattoo Removal Veronica Penza1, Damiano Salerno1,2, Alperen Acemoglu1, Jes us Ortiz1, Leonardo S. Mattos1 AbstractLaser tattoo removal is a standard non-invasive method for removing color pigments on the skin. Increasing number of tattooed people who want to remove their tattoo has driven the medical laser market to develop new technologies for painless, scar-free and complete tattoo removal. However, manual use of such laser systems creates post-operative com- plications since they do not guarantee (i) protection on non- tattooed skin from laser exposure, nor (ii) precise control of the laser focus during the operations for best performance. This paper introduces deTattoo, a robotic system to improve tattoo removal operations. A robotic arm is equipped with a RGB-D camera and a visible laser, in eye-in-end confi guration. A hybrid visual servoing control is proposed to guarantee the correct pose of the laser with respect to the tattooed tissue while compensating body motions. 2D features tracked with a mass-spring-damper deformable mesh model are combined with the 3D reconstruction retrieved from a RGB-D camera in order to build the control law. Several experiments were conducted to evaluate the performance of the system with a fi xed or moving tattooed surface, at different inclinations. Results showed that the proposed framework is able to fulfi l the laser-based tattoo removal requirements, providing high positioning accuracy ( 1mm) orientation ( 0.2) and body motion compensation. I. INTRODUCTION The art of tattoo was already practiced since ancient times in every culture, as a way to express thoughts, feelings and memories through paintings on the body. Its popularity has been continuously increasing in the modern era, such that, according to a survey conducted in 2015 by The Harris Poll, 29% of US adults have at least one tattoo, with an grow rate of 8% with respect to the previous 4 years 1. However, human nature is inclined to changes: one-fourth of people with tattoos regret getting them, and the increasing demand of tattoo removal creates the need for effective techniques. In contrast with different techniques for tattoo removal tested over the years, including surgical excision, dermabra- sion, and chemical destruction, many of which cause damage to surrounding tissue 2, recent advances have led to the use of laser technologies, which has been demonstrated to be more effi cacious and presenting fewer side effects. This technology is based on the concept of selective photother- molysis, a theory described by Anderson and Parrish in 1980 3. Different chromophores (melanin, pigment, water and oxyhemoglobina) absorb different wavelengths of laser light. 1Veronica Penza, Damiano Salerno, Alperen Acemoglu, Jes us Ortiz and Leonardo S. Mattos are with Biomedical Robotics Lab, Department of Advanced Robotics, Istituto Italiano di Tecnologia, via Morego, 30, 16163 Genoa,Italyveronica.penza, alperen.acemoglu, jesus.ortiz, leonardo.demattosiit.it 2Damiano Salerno is with the Universita degli Studi di Catania, Dipar- timento di Ingegneria Elettrica Elettronica e Informatica, Viale A. Doria 6, 95125, Catania, Italydamiano.salernostudium.unict.it When a chromophore is heated for shorter than its thermal relaxation time (i.e. the time required by the tissue to release 50% of its heat after laser irradiation), selective destruction of that chromophore can occur without damaging surrounding tissue. Therefore, the laser absorption by the tattoo granules strongly affect the effectiveness of the tattoo removal. In the current practice, an operator has to manually point the laser on the skin surface to remove the tattoo. One laser pass should be performed over the desired treatment area with minimal ( 10%) overlap between spots 1. Laser treatments are typically preformed at monthly intervals or longer to permit adequate clearing of ink and appropriate skin healing between treatments. Recent studies have shown increased effi cacy of laser tattoo removal when several laser passes are delivered during each treatment session for removal. All these aspects make the removal of a tattoo a complicate process, which most of the time leave the patient unsatisfi ed. The treatment is painful, there could be residual tattoo, scars, and it could take even years to remove a tattoo, affecting the patient physically, psychologically, and socially. This paper introduces deTattoo, an innovative robotic system to autonomously perform laser tattoo removal by automatizing and optimizing the treatment. The system com- bines robotics and computer vision in order to (i) identify and track the unknown tattooed surface area to compensate for body motion, (ii) plan the optimal laser path to avoid tissue thermal damage and (iii) control a robotic arm to accurately perform the laser positioning and tattoo removal. deTattoo is a medical application which has to provide reliability and safety during the tattoo removal process. A vision-based control strategy is necessary to ensure: A constant laser-tissue working distance during all the removal process; Laser perpendicularity to the surface, in order to avoid spot distortion; Compensation of body motion during the operations. To this end, a framework for a hybrid visual servoing approach is presented, where the combination of 2D visual features and 3D features obtained from an on-line recon- structed surface allows the computation of the correct laser trajectory and positioning. The fi nal aim of this system would be to achieve the following benefi ts compared to the current procedure: a reduced number of treatments due to an improvement in the removal strategy and reduced tissue damage due to a precise and automatic laser path planning. The rest of the paper is organized as follows. Section II reviews the state-of-the-art on hybrid visual servoing control. Section III gives a general overview on the system architecture. Section IV presents a 2D deformable mesh 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Macau, China, November 4-8, 2019 978-1-7281-4003-2/19/$31.00 2019 IEEE4461 tracking algorithm based on optical fl ow and mass-spring- damper deformable model for the tattoo motion tracking. The strategy to plan the laser path trajectory is described in Section V, whereas Section VI describes the proposed hybrid visual servoing control. In Section VII, we report the experiments conducted to validate the proposed system and results. Finally, discussion and conclusion are provided in Section VIII. II. BACKGROUND AND RELATED WORK The aim of vision-based control schemes is to defi ne a relationship between the end-effector velocity and changes that occur in the observed object, in order to minimize an error e(t) 4, defi ned as: e(t) = s s,(1) where s is a vector of visual/geometrical features identifi ed on the object, and s contains their desired values. In image- based visual servoing (IBVS)4, s consists of a set of visual features extracted from the image and consequently its error trajectory is defi ned in the image plane. IBVS relies on the online calculation of the image (or feature) Jacobian and on the distance between the target object and the camera, which may be diffi cult to estimate 5. In position-based visual servoing (PBVS) 4, 5, s consists of a set of 3D parameters estimated from image measurements, and its error trajectory is defi ned in the Cartesian frame. PBSV depends on the optical parameters of the visual system and it is very sensitive to calibration errors 5. To overcome these drawbacks, hybrid visual servoing approach was introduced in 6, combining a set of 2D and 3D features in s. This approach combines object pose estimation and image infor- mation, improving the robustness of the control. Keeping into account the aforementioned considerations about deTattoo application, a hybrid approach has been chosen to control the robotic arm based on the information retrieved from the RGB-D camera. In the context of our application, the object shape (i.e., the tattooed surface to scan) is unknown and 2D and 3D features must be detected online. III. SYSTEM ARCHITECTURE deTattoo system is shown in Figure 1. A 6DOF robotic arm is equipped with an RGB-D camera and a laser pointer. An eye-in-hand confi guration is used in order to avoid target occlusion during the execution of the trajectory. The robotic arm is located above the tattooed surface (hereafter for simplicity called object), having the tattoo centred in the camera. The proposed framework consists in: Manual selection, on the RGB image, of the surface area containing the tattoo and automatic segmentation of the tattoo skeleton; Defi nition and initialization of a 2D triangular mesh on the selected area and 2D features detection for tracking purposes; Based on the previous steps, the laser path along the tattoo skeleton is computed in order to maximise the distance between the delivered laser spots; Manual SelectionTattoo Skeleton 2D Mesh and Tracking Path Planning 3D Reconstruction Hybrid Visual Servoing Fig. 1.deTattoo system overview. For each point representing the tattoo, the combination of its position updated at run-time based on the mesh tracking, and the 3D surface provided by the RGB- D camera are exploited for the hybrid visual servoing control. IV. 2D DEFORMABLE MESH TRACKING The operators manual selection of the area containing the tattoo allows to isolate the portion of image to be processed and exploited for visual servoing. The fi rst step consists in the identifi cation of the tattoo skeleton, i.e the 2D points pt(u,v)i, where u and v are the image pixel coordinates and i the number of points the skeleton is made of. These points will be used as 2D features in the hybrid visual servoing approach to position the laser spot for tattoo removal. The desired tattoo skeleton is extracted with a simple thresholding technique. Due to different factors, such as the relative camera-object motion or body motion, ptineed to be robustly tracked in time. At this stage, a set of salient features fhis detected. We used BRISK feature detector 7 since it was suitable for our application. However, simple feature-based tracking technique would not be suffi cient to update pti, since the presence of salient features cannot be ensured along all the tattoo skeleton. Motivated by this, a geometrical surface model has been used to estimate the motion and deformation of the object, providing a deterministic number of control points whose positions is updated based on the tracking of fhfeatures. Consequently, ptiare linked to the geometrical surface control points and updated based on these links. This method can cope with any kind of feature detection and tracking, provide it is robust within the specifi c scenario. In this work, we used a 2D triangular mesh model of N vertices vj= (ux,vy)Twith j = 1.N and expressed as: V = (v1,.,vN).(2) Any point, p, within the mesh can be located through a warping function W(p;V) by using the vertices of the trian- gle it lies within and its barycentric coordinate (bl,bm,bn), where l,m,n represent the triangle vertex indices and where bl+ bm+ bn = 1. The warping function is defi ned as: W(p;V) = ?b 0 0b ? ?v jx vjy?T(3) 4462 vl vmvn vl vmvn kD k kD cDcD cD kf kf kf vl vmvn kD kk MeshInternal forcesExternal forces fh Fig. 2.Mass-spring-damper deformable mesh model. On the left, the regular triangular mesh. In the center, the internal forces (edges and angles) of a triangle. On the right, the external forces produced by the features lying inside a triangle. The deformation model chosen to update the state vector V, and consequently pti, is a mass-spring-damper model 8, as shown in Figure 2. Following this model, each vertex of the mesh vjis subjected of internal Fintjand external forces Fextj, being the total force Fj= Fintj+Fextj. The internal forces try to keep the shape and dimentions of the triangles, while the external forces are produced by the movement of the neighbour features. Inspired by 9, the internal forces have two displacement components: Fedgeand F . The fi rst one is a force between two neighbour vertices viand vjaccording to a spring- damper system, and computed as Fedgeij= Fkij+ Fcij, where Fkij= kD (|vi vj| l0) vi vj |vi vj|, (4) Fcij= ? cD ( vi vj) vi vj |vi vj| ? vi vj |vi vj|, (5) where kDis the linear stiffness constant, cDis the linear damping constant, and l0is the initial length of the edge. The angular force Fproduced between two adjacent edges, defi ned by the vertices (vi, vj) and (vi, vk) respec- tively, is calculated as Fijk= k( 0), where 0is the initial angle, kis the angular stiffness constant and (vi,vj,vk) = arccos ?(v j vi) (vk vi) |vj vi| |vk vi| ?2 ,(6) Being Ejthe subset of neighbour vertices of vj, the internal forces of this vertex are then computed as: Fintj= X kEj Fedgejk+ X i,kEj Fijk,(7) The external forces Fextjare calculated as the sum of the forces applied by each neighbour feature. A neightbour feature of a vertex vj is defi ned as a feature that lays inside any triangle which contains the vertex vj. The force applied by a feature fhis calculated as: Ffjh= kf (|fh vj| ljh0) fh vj |fh vj|, (8) were kfis the stiffness and ljh0is the initial distance between the vertex vjand the feature fh. Consequently, begin Hjthe group of neightbour features, the external force applied to a vertex vjis calculated as follows: Fextj= X hHj Ffjh.(9) The estimation of the mesh deformation is computed by solving the dynamic equation of the system taking into account all the forces applied to each vertex using the explicit Eulers method. V. LASER PATH PLANNING Once the tattoo skeleton points pti are identifi ed and tracked as described in the previous Section IV, the se- quence in which the laser spots are delivered to these points (hereafter called laser path) has to be computed in order to maximise the distance between them so as to prevent tissue thermal damage. The optimal laser path planning has been addressed as the Travelling Salesman Problem (TSP)10. Its most typical representation consists in fi nding, given a set of cities and the distances between each pair of them, the shortest journey that a salesman must follow to visit all the cities one and only once. In our case, the cities are represented by the 2D tattoo points ptiand the problem is to fi nd the path with the maximum distance between consecutive points. TPS can be modelled as an undirected weighted graph, where ptiare the vertices of the graph, the paths between them correspond to the edges, and a paths distance between ptiand pti+1is the edges weight di,i+1. One of the most used approaches is to fi nd all the possible paths that the laser spot could take, in a recursive way. The basic idea in recursive TSP approach is to keep track of the nodes that can be navigated starting from each single node, and then recursively visit them if they have not already been visited along that path. At the end of recursion, a tree-like structure is created. The Held-Karp algorithm 11 is used to solve the TPS. It proposed the bottom up dynamic programming approach as a solution to improve the brute-force method of solving the traveling TPS problem. Since the drawback of applying this algorithm in real application is the exponential time complexity O(2nn2), we decided to apply it to subsets of ptiwith i = 1.m and m = 8, respectively. The ptivector is randomly re-ordered before applying the algorithm in order to create subsets of points in randomized sequence. VI. HYBRID VISUAL SERVO CONTROL As previously mentioned, hybrid visual servoing is based on the defi nitions of both 2D and 3D features, according to the information retrieved from the RGB-D camera. In the context of our application, the object (i.e., the tattooed surface to scan) is unknown and 2D and 3D features must be detected and updated online. 2D features are represented by pti, whose position is linked to the mesh vertices V and updated run-time based on the tracking model described in Section IV. 3D features are extracted from the depth images, 4463 W c c* o Fig. 3.Transformation involved in deTattoo system. as described in details in the following section. For the sake of clarity, Figure 3 shows the transformations involved in the system. A. 3D Features Extraction Considering the 3D reconstruction of the object retrieved from the RGB-D camera, the following 3D features are extracted: Orientation u, expressed as axis/angle parametrization of the rotationcRcthat the camera has to achieve to move from the desired camera frame to the current one. It can be defi ned as: cT c = cT o oT c (10) wherecTois the transformation between the desired camera frame c and the object frame o andcTo is the transformation between the current camera frame c and o, i.e. the current object pose in the camera frame. In order to build u, only the rotational part of matrices in Eq. 10 is taken into account. Desired orientation can be selected as cRo= I3, that is the condition for the perpendicularity between camera/laser (assuming that they are collinear) and the object. Since oT c is time-variant and servo-dependent, it has to be computed at run-time, as described in Section VI-B. Depth defi nes the current depth zcwith respect to the desired depth z c as: z = log( zc z c )(11) B. Online Object Pose Estimation The object posecTowith respect to the camera frame c, is defi ned as follows: 1) The origin of the object reference frame o is placed on the 3D point corresponding to the current selected 2D point pt. 2) z-axis is considered as the normal of the local plane fi tting a subset of 3D points within a window n n around pt; 3) y-axis is the y comp