IROS2019国际学术会议论文集 0479

Vision based magnetic platform for actuator positioning and wireless control of microrobots Azaddien Zarrouk Karim Belharet and Omar Tahri Abstract This work presents a method to guide microrobots by positioning a magnetic actuator using hybrid vision system The used actuator mounted at a robot end effector creates local maxima of the magnetic fi eld magnitude which results in an attractive point for microrobots in its infl uence zone The hybrid vision system serves to control the actuator position through the robotic platform to make the trapped microrobot undergo a planned trajectory In fi rst validation results the particle driving is achieved in open loop with no adjustment with respect to the planned trajectory Such scheme can be used in the case where the microrobot position can not be measured in real time The second validation results deal with the case of microrobot positioning by visual servoing This can be used in the case where recovering microrobot position is possible for example in eye treatment I INTRODUCTION Mobile microrobots have been proposed to be used in several areas ranging from micro manipulation in vitro to in vivo applications thanks to their small scale dimensions that range from micrometers to centimeters and the ability to access small and complex environments The use of external magnetic fi eld has been shown to be a preferred medium for controlling microrobots wirelessly when the integration of the power supply on the micro robot is not possible Control using external magnetic fi eld gradient has many applications in medical fi eld especially in minimally invasive surgery such as targeted drug delivery hyperthermia brachytherapy 1 2 3 4 5 To generate the necessary magnetic fi eld gradient to con trol remotely the microrobots several magnetic actuators have been developed using either electromagnets or perma nent magnets Some works used Helmholtz coils Maxwell coils or various combinations thereof with different con cepts of control such as actuation by exploiting magneti cally resonating structures 6 or exploiting low Reynolds number phenomenon through rotating magnetic fi eld 7 8 Electromagnetic systems with the ability to perform 5 DOF magnetic manipulation have been also developed such as the OctoMag 9 and Magnetecs 10 On the other side for permanent magnets the Stereotaxis NiobeR Magnetic Navigation System uses two massive permanent magnets to control a catheter in the cardiovascular system In 11 12 the authors developed an actuator using two permanent magnets in order to push and guide magnetic micro particles into the cochlea An ability to 5 DOF control arewithINSACentreValdeLoire Universit ed Orl eans PRISMEEA4229 Bourges Franceomar tahri insa cvl fr azaddien zarrouk insa cvl fr is with the HEI campus Centre PRISME EA 4229karim belharet yncrea fr Fig 1 Experimental setup shows the different components of the platform of an untethered magnetic microrobot using a single rotating permanent magnet has been shown in 13 The 3 DOF position control is achieved in closed loop while 2 DOF orientation control in open loop In 14 a new method to achieve full 5 DOF control using permanent magnets that rotate in place is proposed The control of microrobot with the magnetic actuators mentioned above is complex On one side for the electromagnet based actuators the magnetic fi elds gradient is limited by the generated current and the workspace is small compared to the actuator size On the other side the control with a single magnet mounted on a robot manipulator is very delicate because the microrobot can be lost at any moment thus it needs perfect conditions In this paper we present a vision based platform for positioning a magnetic actuator relative to the workspace and guide microrobots The vision system used is a hybrid system composed of a camera and two microscopes allowing to create a link between the macro and micro scale 15 The magnetic actuator is an actuator with four permanent magnets 16 capable of generating a local maximum of fi elds remotely in the plane A maximum of local fi elds means a convergence point of microrobots This property allows control of microrobots remotely with precision and much less complexity The presented method allows both to position the actuator to correct the position of the workspace assuming that it can move during the tracking of the trajec 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 IEEE1601 y z M x z 0 Mi c r o r o b o t i n i t i a l p o s i t i o n P i 1 P i 2 P i 3 P i 4 F i n a l p o s i t i o n P f A c t u a t o r X Z p l a n e A c t u a t o r Y Z p l a n e Fig 2 The four magnet system and the area around its convergence point in xz plane Black dots are microrobots in different positions and yellow narrows are the trajectories released by the microrobots because of the effect of the convergence point The scheme in the bottom right corner represents the actuator design in the yz plane The force is calculated for microrobot with radius of 250 m and magnetisation kMk 9 46 105A m 1 tory and visual servoing of the position of the microrobot in the organs where the microrobot position is accessible like the eye for example The four magnet based actuator and the hybrid vision system with a motorized XYZ stage represent a promising solution that is close to the end use and offers both open loop and closed loop control The remaining of this paper is organized as follows the next section gives a brief explanation of the four magnet based actuator and the hybrid vision system In Section III provide a method for vision based actuator positioning and microrobot steering strategy Finally section IV shows the experimental results The paper is concluded in section V II WIRELESS ACTUATOR AND HYBRID VISION SYSTEM In this section we will give a brief explanation of the mag netic actuator the hybrid vision system and the properties of each system A Four magnet based actuator A previous work on shaping the magnetic fi eld to easily control a microrobot have led to design a four magnet based actuator capable of wireless position control of an unteth ered microrobot in 2D workspace even in the presence of reasonable perturbations 16 Its particularity is the creation of local maximum of magnetic fi eld magnitude at a distance This maximum acts as a point of convergence for magnetic microrobots In fact because the magnetic force is always in the direction of the gradient of the magnetic fi eld norm Fig 3 The hybrid vision system 15 with the various unknown homogeneous transformations between sensors coordinate systems the microrobot will be attracted to the local maximum when it is positioned in the infl uence zone of this maximum Fig 2 shows the magnetic force direction and strength in the area around the convergence point The trajectory of microrobots placed in different initial positions is shown in this area to demonstrate that without any control efforts the microrobots converged to a known position which is the actuator convergence point When mounting the system on robot manipulator end effector or any XYZ positioning stage the stabilization point can be placed at any desired position using simply translational motions It ensures open loop control of the microrobot when evolving in confi ned environments where visual sensors are useless Besides if the organ is accessible and the microrobot position can be measured in real time using vision sensors visual servoing can be used in such case for more accurate control B Hybrid vision system The hybrid vision system as shown in Fig 3 consists of a camera and two microscopes The camera is used to control the position of magnetic actuator The two microscopes are used to recover the 3D position of the microrobot and the workspace The hybrid vision system allows real time track ing of magnetic actuator the workspace and the microrobot when it is possible In 15 the proposed calibration method of this hybrid system uses two patterns rigidly attached It allows creating a link between micro and macro scale to both control microrobots and position the actuator Positioning accuracy has been studied in 15 using a laser mounted on robot manipulator to target points in the fi eld of view of microscopes The results showed that the system was able to estimate the needed laser pose to target the desired point with an error less than 500 m Next section is dedicated to detail the actuator position ing method and microrobot control strategy using the two 1602 components presented in this section III VISION BASED ACTUATOR POSITIONING AND MICROROBOT STEERING In this section we will discuss the full process of mag netic microrobot steering including the actuator positioning the workspace movement adjustment using the calibrated vision system and fi nally the microrobot steering in planar workspace with the four magnet based actuator The full process is shown in Fig 4 Initially the microrobot trajectory is planned in the micro scale workspace frame The trajectory is discretized and mxd xd yd 0 Tis the vector contents the coordinates of the desired trajectory in the micro scale workspace environment We defi ne the actuator workspace frame with respect to the microrobot workspace because the actuator generates a convergence point at distance whose its position is given by the vector actxc xc 0 0 T mTact w I 3 3 actxc 01 31 1 For a given trajectory the actuator will release the same movement of the microrobot but in its workspace frame Therefore for each point of the planned trajectory the position of the actuator in its workspace frame mTact w is given by the same vector xd The corresponding desired relative pose actwTact is expressed as follows actwTact I 3 3 mxd 01 31 2 The camera creates a link between the frame of the robot and that of the microscopes thanks to the already calibrated hybrid vision system and the robot camera calibration step The last calibration problem is classical problem in robotics fi eld that have been suffi ciently studied 17 18 With all the necessary transformations see Fig 5 obtained the planned trajectory is transformed to the robot manipulator For each desired position of the microrobot the robot end effector pose robTef f is expressed as follows robTef f robTmicmicTmmTact w actwTactactTef f 3 where robTmic robTcamcamTmic the relative pose of the mi croscopes frame with respect to the robot base frame actTef f the relative pose of the robot effector with respect to the actuator frame micTm the relative pose of the microscale workspace with respect to the microscopes frame It is obtained with the micro stereoscopic system From all the relative poses in equation 3 micTm is the only homogeneous transformation that is updated in real time As seen in Fig 4 camera can be selected to track markers on workspace In this case micTm is calculated as follows micTm micTcamcamTMMTm 4 Mi c r o r o b o t mo v e me n t p l a n n i n g T r a j e c t o r y a d j u s t me n t Wo r k s p a c e Mo v e me n t d e t e c t i o n C a me r a R o b o t ma n i p u l a t o r A c t u a t o r C l o s e d l o o p C l o s e d l o o p O p e n l o o p A c t u a t o r p o s i t i o n i n g Mi c r o r o b o t g u i d i n g Mi c r o s c o p e s Fig 4 Microrobot steering strategy using the hybrid vision system and the four magnet actuator r o bT c a m mi cT m mT a c t w e f f T a c t r o bT e f f c a mT mi c Fig 5 Different relative poses between the vision based platform components where camTM is the pose of the markers in the camera frame and MTm is the relative pose of the micro workspace frame with respect to the markers 1 Vision basedactuatorpositioningandmicrorobot steering strategy From equation 3 we see that positioning the actuator with respect to the workspace is ensured in real time with the calibrated hybrid vision system However once the planned trajectory is defi ned the positioning is achieved in open loop with no feedback on the micro robot location Therefore the hybrid vision system serves only for positioning the actuator in the workspace This feature is im portant in medical fi eld to take into account the movements of the patient during the treatment For example to guide microrobots inside the inner ear cochlea the trajectory is planned initially to be released in open loop However 1603 once the patient head moves the planned trajectory should be corrected with respect to the new position of the head On the other hand the platform can track and correct the microrobot position when it is accessible for visual sensors see Fig 4 like the case of the eye treatment In this case we estimate the microrobot position with the microscopes the actual position is compared to the desired position and corrected to minimize the error 2 Positioning correction using the convergence point of the actuator In equation 3 the relative pose actTef f of the actuator with respect to the effector is obtained using the actuator CAO model In the experimental platform this relative pose may not coincide perfectly with the CAO model Errors may come from the 3D printing of the actuator and when mounting the actuator on the effector In the following we will estimate these errors using the actuator convergence point The actuator is moved manually to place the microrobot in the workspace whose position mxp xp yp 0 Tis given by the stereoscopic system We replace the desired position mxd with the microrobot actual position mxp in equation 2 actwTact I 3 3 mxp 01 31 5 The experimental relative pose actTef f is obtained with the following equation actTef f actwT 1 act mT 1 actwmicT 1m robT 1microbTef f 6 where robTef f is given by the inverse geometric model of the robot actwTact is calculated with the equation 5 The other relative poses have been kept constant To avoid any measurement errors multiple actTef f are calculated for different robTef f and actwTact IV EXPERIMENTAL VALIDATION This section shows the experimental results of the posi tioning method with the hybrid vision system Firstly the results of the correction using the point of convergence of the actuator are shown Then two types of experiments were con ducted to demonstrate the feasibility of the robotic platform with the positioning method to control the microrobot in both closed environments cochlea and others that are accessible for visual feedback the eye for instance As shown in Fig 1 the used prototyping platform is composed by a magnetic actuator camera and two microscopes hybrid vision sys tem and robot manipulator to perform the positioning of the actuator The workspace has circular shape Fig 6 fi lled with viscous liquid composed by a mixture of 80 water and 20 glycerol with dynamic viscosity of 17 288 10 4 Ns m2 The microrobot is spherical magnetic particle with radius of 250 m 1 Positioning correction using the convergence point of the actuator To obtain the position of the convergence point with respect to the robot end effector the CAO model of the actuator and the model of the four magnets were fi rstly used As predicted errors due to fabrication of the magnets TABLE I CORRECTION RESULTS OF THE RELATIVE POSEEFFECTOR ACTUATOR x y z mm degrees Before correction 75 307 75 307 57 45 0 0 After correction 66 521 68 197 65 837 48 331 5 647 0 191 Mi c r o b o t 5 mm Ma r k e r s Fig 6 The workspace where the microrobot is controlled The markers placed on the workspace are used by the vision system to detect its movement container and the actuator attachment to the end effector affected the actuator positioning with respect to a target point under microscopes The errors comes from vision system are already studies and has been proven to be under 500 m The results of the relative pose Effector Actuator correction are shown in table I These results show that the error on the position along the x axis is up to 9mm and on the orientation up to 5 degrees on the y axis Which shows the interest of this correction step of the relative pose effector actuator estimation 2 Vision basedactuatorpositioningandmicrorobot steering strategy The fi rst experiment was performed as suming that the position of the microrobot is inaccessible However we can observe the workspace from the outside and so we can detect its movements The planned microrobot trajectory is followed in open loop in the micro workspace with no feedback On the other hand the actuator position is adjusted based on the image of pattern mounted on it using the camera The scenario considered in this experiment is similar to the case of the internal ear the control of microrobot is done in the cochlea but markers can be placed on the head of the patient to detect its movements Here we have two options to follow the workspace either with the camera by attaching markers that are visible by the camera to the workspace Alternatively with microscopes by gluing small markers directly onto the workspace We chose the second option We release two trajectory types rectangular and spiral trajectory We predefi ne the trajectory of the microrobot and then we transform it to the robot manipulator frame Finally we start guiding the microrobot along the predefi ned trajectory The results are shown in Fig 7 a and Fig 7 b at a moment during the navigation we simulate the workspace disturbances by moving the 1604 2 mm P e r t u r b a t i o n a b Mi c r o r o b o t t r a j e c t o r y Mi c r o r o b o t t r a j e c t o r y P e r t u r b a t i o n c d Fig 7 The pictures present the results of the open loop trajectory tracking and closed loop workspace position correction and the error results between the predefi ned and the experimental trajectory respectively a and c for a rectangular trajectory b and d for a spiral trajectory workspace to check the trajectory correction using the vision system Fig 7 c and Fig 7 d show the norm of the error between the predefi ned and the experimental trajectory The peaks represent t