IROS2019国际学术会议论文集 0501

An assisted telemanipulation approach combining autonomous grasp planning with haptic cues Maxime Adjigble Naresh Marturi Valerio Ortenzi Rustam Stolkin Abstract This paper presents an assisted telemanipulation approach with integrated grasp planning It also studies how the human teleoperation performance benefi ts from the in corporated visual and haptic cues while manipulating objects in cluttered environments The developed system combines the widely used master slave teleoperation with our previous model free and learning free grasping algorithm by means of a dynamic grasp re ranking strategy and a semi autonomous reach to grasp trajectory guidance The proposed re ranking metric helps in dynamically updating the stable grasps based on the current state of the slave device The trajectory guidance system assists in maintaining smooth trajectory by controlling the haptic forces A virtual pose controller has been integrated with the guidance scheme to automatically correct the end effector orientation while reaching towards the grasp Various experiments are conducted evaluating the proposed method using a six degrees of freedom dof haptic master and a seven dof slave robot Results obtained with these tests along with the results gathered from the performed human factor trials demonstrate the effi ciency of our method in terms of objective metrics of task completion and also subjective metrics of user experience I INTRODUCTION Robotic telemanipulation plays a signifi cant role in ac complishing a variety of challenging tasks in various fi elds It enhances human manipulation capabilities in remote and safety critical environments where direct human interaction with the scene objects is not considered feasible For in stance telemanipulation systems are used in the medical fi eld to execute complex surgical procedures 1 2 and in nuclear settings they help in carrying out complex de commissioning tasks 3 Most of the currently used tele manipulation setups in unstructured environments heavily rely on operators experience Nevertheless executing remote manipulation tasks by means of mechanical devices in such environments is highly challenging and tedious Moreover the cognitive load on the operator is high 4 For example to pick and place an object in a remote scene the operator needs to make wise guesses for the inverse kinematics avoiding mechanical singularities while manoeuvring the robot towards a speculated target grasp position We strongly believe that the operator workload can be signifi cantly re duced through sensor guided assistance To this end we present an assisted telemanipulation method that combines All authors are with the Extreme Robotics Laboratory University of Birmingham Edgbaston B15 2TT UK Email m k j adjigble n marturi v ortenzi r stolkin bham ac uk This work was supported by the UK National Centre for Nuclear Robotics part funded by EPSRC EP R02572X 1 It was also partly sup ported by H2020 RoMaNS 645582 and the Faraday Initiative project ReLiB project Rustam Stolkin was supported by a Royal Society Industry Fellowship b a Fig 1 Illustration of the proposed assisted telemanipulation approach a Human operator performing the manipulation task by moving the haption device along a generated virtual path while viewing the scene in camera views inset Robot end effector being guided to the target grasp i e the grasp with maximum ranking Solid blue line represents the generated zero force path Tool orientation alignment has been automatically performed while the position is controlled by the master movements b Robot grasping an object from clutter haptic cues with an advanced grasping algorithm 5 Such an approach enables the operator to focus more on the task being performed whereas all the kinematic computations and trajectory planning to the grasp locations are automatically handled by the system One of the earliest methods for assisted telemanipulation was virtual fi xtures 6 which works by overlaying the sensory information on operator s workspace Later Bettini et al 7 have extended it by designing control laws for robot tool positioning and curve following Besides many recent works have used virtual fi xtures for telemanipulation via shared control 8 10 Despite being an effective tool for assisted telemanipulation the main drawback of virtual fi xtures is that they are hard to generalise and require tedious pre programming To the point haptic based teleoperation has well been explored in the fi elds of medical robotics 1 robotic rehabilitation 11 etc A common verdict is that the haptic information allows faster refl exes than the visual feedback and hence results in better performances than using visual cues alone in attaining tasks with high cogni tive load 12 Although haptic cues enhance the operator telepresence in performing basic teleoperation manipulation tasks overall task performance especially in case of unstruc tured unknown environments still greatly depends on human decisions Several recent papers have suggested to incorpo rate autonomous functionalities with direct user control in or der to improve the telemanipulation performance A method integrating semi autonomous grasping with a telepresence system was presented in 13 where the operator selects 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 Crown3164 Acquire point clouds Build scene model LoCoMo Grasp generation Grasp ranking Master slave coupling Master force generation Haptic master VisionGrasp Target grasp pose slave pose Fig 2 Illustration of our assisted telemanipulation pipeline The three components i e the vision in red box grasp in green box and teleoperation in blue box can be seen The grasp component takes the input model built by the vision component and generates a set of grasp hypotheses for the entire scene The most stable grasps from this set are selected using our LoCoMo metric The selected stable grasps are re ranked using 13 based on the feedback from the teleoperation component The best grasp of the re ranked ones is executed on the slave robot the target gripper pose based on the information gathered from independent contact regions A semi autonomous tele manipulation system in which the slave robot autonomously executes the task after understanding the operator s intent was presented in 14 An assisted telemanipulation method based on the reconfi gurable stack of tasks was presented by Stoyanov et al in 15 Some other related work includes 16 19 While these shared control methods greatly as sist in creating advanced telemanipulation setups appropriate attention is required while blending autonomy with direct human control 18 On a side note until now a lot of research has been realised in developing advanced grasping and robotic motion planning methodologies We suggest the reader to refer 20 22 for further details A shared control method for master slave manipulation and grasping was previously presented by Ghalamzan et al in 23 This work closely resembles our approach It works by selecting the best grasp candidate for a specifi c object in order to minimize the joint velocities during the post grasp phase During the teleoperation phase a force feedback is generated on the haptic device in order to guide the operator to the best suitable pose i e the pose which results in optimal velocities maximum manipulability In contrast the stable grasps that are generated using our model free method 5 are dynamically updated during the teleoperation phase in our proposed approach The update process depends on the current pose and reachability of the slave robot and the grasp likelihood of an object in the scene Unlike in 23 our method deals with a cluttered scene consisting of multiple objects in which case it can dynamically update the grasp poses for different objects present in the scene based on the current position of the slave This will assist the operator in multiple ways i since the grasps are dynamically updated no manual selection of the stable grasps is required i e the task of grasp selection is reduced to naturally moving the slave device close to the desired grasp location ii the cognitive load of the operator is reduced due to the virtual guidance on the desire grasping trajectory and iii problems associated with the spatial awareness caused due to lack of direct depth perception are also diminished The major contributions of this work are i We integrate our previously presented robust model free and learning free method for grasp candidate generation with force guided teleoperation The method generates and ranks multiple grasp candidates irrespective of the object model based on the metric designed with the local contact moment features LoCoMo ii We propose a metric to dynamically re rank the grasp candidates to propose the most stable and closest grasp during the teleoperation mode This metric has been de signed by combining the distances between the grasps and the robot arm and the LoCoMo score iii Next we implement a virtually guided shared control where upon selection of the best grasp haptic forces are generated to maintain a smooth trajectory of the slave robot to the target grasp location Natural movements of the hand are directly mapped to the corresponding movements for the slave robot iv Finally we implemented a shared pose controller for orientation alignment where the translation control is performed via master and the rotations are automati cally updated to attain a stable reach to grasp position during the teleoperated trajectory As shown in Fig 1 a inset The rest of the paper is organised as follows the fundamental components of our approach are explained in detail in Sec II Our approach to assisted telemanipulation is presented in Sec III Experimental results analysing the effectiveness of our approach and human factor study are reported in Sec IV II FUNDAMENTAL COMPONENT DESCRIPTION In this section we present the prerequisites for our assisted telemanipulation approach As mentioned our method is 3165 developed by blending dynamic grasp candidate selection with haptic guidance Its pipeline is shown in Fig 2 which is composed of four inter linked components We will present each component of the pipeline individually in the following and show how they are combined together in Sec III to per form effi cient telemanipulation in unstructured environments A Master Slave teleoperation Master Slave teleoperation is a well studied topic in robotics It can be achieved in various ways from a simple direct mechanical coupling between master and slave 24 to a more advanced virtual coupling with force feedback 25 In this work our main focus is on virtual bilateral mas ter slave coupling where the forces are generated virtually i e via software Besides a second generated force feedback is added to these virtual forces when the assisted guidance is enabled Let Fsand sbe respectively the desired slave Cartesian end effector forces and desired slave joint torques They can be computed as in 1 and 2 Fs Ks Xm Xs Kd Xm Xs 1 s JTFs 2 where Xs Xs are respectively the Cartesian pose and velocity of the slave robot and Xm Xm are the Cartesian pose and velocity of the haptic master device respectively Ksand Kdare adjustable gains controlling the stiffness and damping of the master slave coupling JTis the transpose of the Jacobian matrix J of the slave robot The forces generated by 1 enable the slave robot to closely follow the movements of the master device At this stage we only have a unilateral master to slave coupling as the movements of the slave robot do not have any effect on the movements of the haptic master device In other terms no force feedback is provided by the master device to the operator No external contact between the slave robot and its environment nor additional forces occurring while grasping objects could be felt by the operator thus forcing the operator to develop an additional awareness of contacts by only visual cues Applying the reciprocal force defi ned by 3 to the haptic master device allows to provide to the operator a force feedback encoding the interaction of the slave robot with it s environment Fm Fs Ks Xm Xs Kd Xm Xs 3 In a mechanical analogy the bilateral master slave cou pling can be envisaged as a virtual spring and damper connection between the end effector of the master device and slave robot as depicted in Fig 3 This scheme allows to control simultaneously the position of the slave robot while providing a force feedback to the master device in case of external contact with the environment Contrary to using joypad controllers for teleoperation which only provide unilateral coupling this bilateral control scheme has been proven to be more effective in practice 25 Master device Slave robot Virtual link Fig 3 Illustration of a bilateral master slave coupling B Grasping LoCoMo metric Robotic grasping is still an open research problem and several methods have been proposed in the literature to tackle this challenge We have recently proposed a model free and learning free grasping algorithm in 5 that is reliable with a high success rate and achieves near real time performance The method is based on the computation of the Local Contact Moment LoCoMo metric and the ranking of grasps based on the product of integrals of LoCoMos for each fi nger of a gripper In Sec III A we extend the grasp ranking for dynamic grasp selection in a teleoperation scenario For two discrete surfaces 1and 2contained in an Euclidean sphere of radius the LoCoMo metric can be computed by 4 as described in 5 Straightaway 4 can be reduced to 5 C 1 max x x 0 0 max x x 0 4 C 0 max x x 0 5 where represents the multivariate Gaussian density func tion given by x 1 p 2 n exp 1 2 x 1 x 6 where x Rn is the covariance matrix and n is the space dimension e g n 3 in R3 max x x 0 is the maximum value of the Gaussian density function centered in 0 with covariance It is known to be equal to 2 n 1 2 thus further simplifying 5 to 7 C 2 n 1 2 0 7 where is the error of zero moment shift vectors of each discrete surface defi ned as n1 n 2 8 nj 1 Nj Nj X i 1 xj i xj j 0 1 9 where xjis the point from 1or 2at which the zero moment shift vector is computed Njis the number of points in jand xj i i 1 N j represents the list of points of j In 8 n1 and n2 need to be expressed in the same reference frame for the error to have a physical meaning 3166 The LoCoMo metric provides an indication on the local similarity between the shape of the gripper used and the local surface of the objects to grasp The highest the score the higher the shape similarity is in terms of their zero moment C Grasping hypothesis generation The problem of grasp hypothesis generation is formulated as the optimization problem of fi nding a set of kinetically feasible fi nger poses that maximize the Contact Moment CoMo metric for each fi nger The CoMo is computed by discretely integrating the LoCoMo metric over the surface of the fi nger using 10 Grasps are formed by selecting nf kinetically feasible fi ngers and are ranked as in 11 Ci 1 Ns n X i 1 Ci Xi 10 R k nf Y i 1 Cwi i 11 where n is the number of points in the vicinity of the fi nger Nsis a normalizing term representing the maximum number of points in the vicinity of the fi nger Ci Xi is the local contact moment probability between a point on the point cloud and its orthogonal projection on the surface of the gripper k is a normalizing term and wiare weights satisfying Pn i 1wi 1 III PROPOSED ASSISTED TELEMANIPULATION METHOD In this section we explain the three important components of our proposed assisted telemanipulation framework i e grasp re ranking force guidance and automatic orientation alignment Selecting grasps on the fl y while teleoperating the robot can be accomplished in different ways In this work we follow a nearest grasp re ranking strategy that has been shown to exhibit effi cient grasp selection in practice and is well suitable for a wide range of applications A Grasp re ranking Given a set of N grasps G which is expressed as G Hj Rj Hj SE 3 Rj R j 1 N 12 where Hjis the homogeneous matrix representing the pose of the gripper and Rjis the ranking value computed for the grasp We introduce dj R some distance between the target grasp pose Hjand the current gripper s pose H In this work the Cartesian distance between the current gripper pose and the target grasp pose has been used However this can be generalized in a straightforward way to take into account the rotations by computing djas the magnitude of dual quaternion error between H and Hj where each homogeneous transform is represented by a dual quaternion More details can be found in 26 We reformulate the grasp ranking equation 11 for a grasp gj G as R 0 j dmax dj dmax dmin Rj 13 Fig 4 Generated grasp hypotheses for multiple objects in the scene placed as a clutter The gripper pose in green represents the highest ranked grasp based on Eq 13 whereas the remaining candidates are shown in wireframes Coloured points represent LoCoMo features where dminand dmaxare respectively the minimum and maximum distances of dj j 1 N The ranking Rj is multiplied by the distance djnormalized over the set of distances dj j 1 N Fig 4 shows the highest ranked grasp green solid among a set of grasp candidates wirefrmes The new metric R 0 j favours grasps that are at the same time close to the current end effector s pose and have a good overall LoCoMo score B Virtual force guidance Upon selecting the highest ranked grasp a collision free trajectory can be computed between the current pose of the robot and the target pose While an automatic motion could be executed to grasp the selected object providing an alternative force feedback guidance along the computed trajectory is especially relevant in case of safety critical applications where a human operator is required in the loop The aim of the virtual guidance is on top of the