IROS2019国际学术会议论文集 2408

Learning Based Robotic Bin picking for Potentially Tangled Objects Ryo Matsumura1 Yukiyasu Domae2 Weiwei Wan1 2 and Kensuke Harada1 2 Abstract In this research we tackle the challenge of picking only one object from a randomly stacked pile where the objects can potentially be tangled No solution has been proposed to solve this challenge due to the complexity of picking one and only one object from the bin of tangled objects Therefore we propose a method for avoiding the situation where a robot picks multiple objects In our proposed method first grasping pose candidates are computed by using the graspability index Then a Convolutional Neural Network CNN is trained to predict whether or not the robot can pick one and only one object from the bin Additionally since a physics simulator is used to collect data to train the CNN an automatic picking system can be built The effectiveness of the proposed method is confirmed through experiments on robot Nextage and compare with previous bin picking methods I INTRODUCTION Bin picking is a common task in the manufacturing processes where typically a human worker is in charge of unloading one part from the bin at a time to feed a machine Automation of bin picking using industrial robots is then crucial on the one hand to relieve workers hands from this tedious monotonous and potentially dangerous task or on the other hand to remove the necessity of implementing any parts feeding machines to arrange the objects to be picked by a robot However robotic bin picking presents a tough challenge that in volve looking selecting and picking randomly oriented objects The topic of solving the bin picking task with a robot has been extensively researched such as in 1 8 nevertheless it remains as a difficult and complex challenge One of the difficulties of the task is due to the complex physical phenomena of the contact between the objects and the robot fingers Hence the success rate of a picking task is usually between 80 and 90 even for simple shaped objects 5 9 10 Let us consider the picking task from a bin full of randomly mixed and staked parts There are two par ticular cases where the contact dynamics between the robot fingers and the objects makes robotic bin picking difficult First while a multi fingered gripper approaches the target object within the pile one of the fingers may con tact a surrounding object In this case if the surrounding 1Graduate School of Engineering Science Osaka University Japan 2Artificial Intelligence Research Center National Institute of Advanced Industrial Science and Technology AIST Japan Fig 1 Robotic bin picking The robot picked the target object but mistakenly picked up more objects since they are tangled object does not move after making contact with a finger it is impossible for the gripper to grasp the target object In the second case depending on the object s shape randomly stacked objects may be entangled with another object as shown in Fig 1 In this case a careful selection of the object is required to guarantee that only one is picked If a robot picks an object entangled with other objects the robot s gripper may unexpectedly pick more than one object at the same time In the worst scenario the overall picking task will fail because the target object is tangled with other parts in such a way that lifting it becomes impossible Especially the second case has not been considered in conventional researches on robotic bin picking To cope with the above two problems at the same time we propose a new learning based approach for robotic bin picking To pick a complex shaped object from the pile the robot has to carefully select and pick an object such that the picked object is not entangled with another object In our proposed method we first generate candidates of gripper s grasping poses by using the graspability index 5 where we can avoid a finger of a gripper contacting a surrounding object while the gripper approaches to the target object The graspability generates a 2D map of the pile indicating that fingers of a gripper can be inserted in between the objects without making collision However just by using the graspability index we cannot avoid a picked object being entangled with another object Hence we introduce the learning based approach by using a Convolutional Neural Net work CNN While conventional learning based methods on robotic bin picking just predict whether or not a robot successfully pick an object from the pile we propose a 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 IEEE7984 Binary image Pixel wise AND Reverse Contact Collision Gaussian Graspability map Peak Detected position Not peak Depth image Not peak Not peak Fig 2 Overview of the graspability calculation novel approach where a CNN predicts whether or not a picked object is entangled with another object We show that a robot can pick a complex shaped object from the pile with higher success rate compared to the conventional CNN based robotic bin picking methods Since collecting enough number of training data used for CNN through the real world experiment is not easy especially for detecting if a picked object is entangled with another object we use a physics simulation of picking an object from randomly stacked pile We show that even if we collect the training data through physics simulation the success rate of a picking task by using our proposed method is high enough as we will discuss in the section of experiment The rest of this paper is organized as follows After discussing the related works in Section II we will discuss our proposed method in Section III In Section IV we will show several experimental results to confirm the validity of the proposed approach II RELATED WORK So far research on robotic bin picking has been mainly done on image segmentation 1 pose identification 2 4 and picking methods 5 6 8 9 11 14 Among them the research on bin picking method can be fur ther categorized into three categories i e the picking methods based on the pose identification of piled objects 4 6 8 the methods without using the pose iden tification 5 and the learning based methods 9 14 As for the methods based on the pose identification of piled objects 4 6 8 we need to carefully set several parameters used in both visual recognition and grasp planning corresponding for each picked object When a robot has to pick several different objects it is not easy for setting parameters for all objects to be picked On the other hand as for the method without using the pose identification Domae et al 5 proposed a method for determining the grasping pose of an object directly from the depth image of the pile based on the graspability index However it is impossible by using this method to avoid a robot picking an object entangled with another object As for the learning based methods Levine et al 11 proposed an end to end approach by using deep neural network while this method requires extremely large number of training data obtained from real world experiments Recently there are some trials on reducing the effort to collect a number of training data by using a method so called GraspGAN 14 cloud database 12 15 17 and physics simulation 10 However the con ventional learning based approach on robotic bin picking just predicts whether or not a robot can successfully pick an object from the pile and have not explicitly considered the case where a picked object is entangled with another object Hence if a robot picks a complex shaped object a picked object may be entangled with another object On the other hand our proposed CNN is used to just predict whether a picked object is entangled with another object The learning approach has also been used for grasping a novel daily object placed isolated on a table 18 21 and for warehouse automation 22 23 Pas et al 20 developed a method for learning an antipodal grasp of a novel object by using the SVM Support Vector Machine Lenz et al 19 used deep learning to detect the appropriate grasping pose of an object Zeng et al 22 proposed a learning based picking method used for warehouse automation As for the manipulation of potentially tangled object some researchers such as 24 25 researched the robotic manipulation of a rope On the other hand this research is the first trial on picking an object from the tangled pile III PROPOSED METHOD This section explains our proposed method for picking only one object from the bin with potentially tangled objects We assume a parallel jaw gripper to pick an object from the pile In our proposed method some candidates grasping poses are first detected using the graspability index 5 These candidates are guaranteed to avoid collision of the robot s finger with the nearby objects while the hand approaches the target object Next we explain the physics simulation used to train our proposed CNN Our CNN is used to predict whether 7985 or not the robot can pick only one object from the bin with potentially tangled objects Finally the grasping pose is selected based on the harmonic mean of the graspability index and the CNN prediction The grasping pose detected in these procedures ensure that a robot can pick only one object from the tangled pile with a high success probability This section details the approach followed in this work A Graspability Index The graspability 5 is calculated by using two mask images one describing a contact region that should be filled by a target object for stable grasping and the other describing a collision region that should not be filled by other objects to avoid collisions during grasping To calculate the graspability two fingers of a gripper are represented by the cross section in the 2D horizontal plane The graspability is computed by convolving the mask images with binarized depth maps which are thresholded differently in each region according to the minimum height of the 3D points in the region and the length of the gripper After computing the region of contact without collision of the gripper and object the graspability map for the gripper model is then obtained by adding a blur with a Gaussian filter As a result the portion of the map with higher pixel value is regarded as more stable These processes are as shown in Fig 2 The grasping poses obtained are guaranteed that there is no collision between the hand and the obstacles and that target object exists in the trajectory of closing the gripper For our method the top five grasping poses detected with the graspability index method were used as grasp candidates B Physics Simulator Our proposed method is based on supervised learning so in order to predict whether or not the robot can pick only one object from a bin with randomly placed and possibly entangled objects labeled training data is necessary More specifically the goal of our CNN is to predict whether or not an attempt to pick an object will result in one and only one object being picked We collect training data using a physics simulator The basic idea for collecting training data is within the simulation the robot attempts to pick an object from a randomly gener ated pile of objects then the number of objects picked is recorded Therefore we needed to build a simulator that could tell the number of objects picked at the end of each picking attempt However first we have to define how to judge the number of objects picked in a single picking motion in the simulator more than one object picked would mean that the target object was entangled with nearby objects To define the entanglement in the simu lator we follow the reasoning described below First if the entanglement is defined as just the number of objects that a parallel jaw gripper successfully picked up then the tangled objects that may drop during the motion are Fig 3 Dataset collection with physics simulation not taken into account So instead of considering only the number of objects picked in by the end of a single picking motion we defined the entanglement as whether or not the objects near the target object moved upward during the picking motion A threshold determines where the part was lifted or not However when several objects were strongly tangled with each other they were not lifted upwards at all due to the combined weight So in this case the entanglement could not be detected So finally we changed the specification of the simulator to automatically lift the object existing at the grasping position without trying to simulate the grasping with the robot gripper Based on the above the specification of the simulator is as follow First our simulator generates a bin with a pile of randomly placed objects and searches for a grasping pose based on the graspability index as shown in the top left of Fig 3 Second the target object the part that is in the closing trajectory of the parallel jaw gripper is automatically lifted At this time this object turns green as shown in the top right of Fig 3 Then if objects excluding the green one move more than a certain amount in an upward direction they turn red as shown in the bottom of Fig 3 Finally our simulator records the picking as tangled if the number of picked objects during a single picking motion is more than one Otherwise if only one object is lifted up it records the picking as not tangled In addition if several objects exist in the closing trajectory the gripper will grasp them together So if there are two or more objects detected in the trajectory of gripper closing the case is immediately recorded as tangled regardless of whether or not other nearby objects move upward By repeating this cycle we can collect a lot of training data labeled with tangled or not tangled during a single picking motion 7986 Conv 1 Pool 1 Fully conn 1 Conv 2 Conv 4 Conv 5Pool 2 Pool 3 Fully conn 2 Fully conn 3 Not tangle Tangle Conv 3 Fig 4 Architecture of the convolutional neural network used for classifying whether a candidate grasping pose will result in the target object being picked without being entangled with another object C Convolutional Neural Network The CNN s job is to predict whether or not the robot will be able to pick one and only one object from a pile of potentially tangled objects The input given to the CNN is a depth image of the top view of the bin The archi tecture of CNN is inspired by the AlexNet 26 network which accomplished significant results in ILSVRC Im ageNet Large Scale Visual Recognition Challenge 27 Here the input to CNN is a rotated depth image so that the orientation of the robot hand is always horizontal The reason for this is that the learning result was better when inputting only a depth image rotated according to a hand than when using both of them separately Our CNN is composed of a series of convolutional layers followed by pooling layers as shown in Fig 4 A max pooling of 2x2 pixel is applied in each pooling layer The result is then processed by three fully connected layers that classify whether or not the robot can pick only one object for the given grasping candidate These classes are denoted by y0 Not tangled and y1 1 y0 Tangled by using the following softmax function yk eak n i 1ea i 1 where akdenotes the weight of the input of the last fully connected layer An activation function is used in the convolution layers and the fully connected layers to avoid the problem of gradient loss For this reason we use a ReLU Rectified Linear Unit function as the activation function f x max x 0 2 In our proposed method the candidates of grasping poses detected by using the graspability index were used as the input to the CNN Finally the grasping pose is selected based on the harmonic mean of the graspability index and the CNN prediction IV EXPERIMENT A Setup In order to evaluate the proposed method two dif ferent experiments were conducted with the following equipment an industrial robot Nextage from Kawada Robotics Inc a depth sensor a parallel jaw gripper and Depth sensor Parallel jaw gripper Randomly stacked pile Fig 5 Experimental setup Depth sensor with a fixed top view of the bin parallel jaw gripper and a bin filled with randomly placed objects a bin with randomly placed objects The setup is shown in Fig 5 First we consider testing our method on several items a different CNN was trained for each item The task of the robot was to pick one object from a pile full of that kind of objects This approach was followed because we consider that whether or not the robot can pick only one object from the tangled pile strongly depends on the shape of each object In this first experiment the proposed method is compared to the picking method that only uses the graspability index 5 and our previous method using a CNN for discriminating success failure of picking 10 For testing 15 items of four different types of objects were used as shown in Fig 6 In the physics simulator we use the 3D models of the objects as shown in the right column of Fig 6 The simulation was used to collect a dataset of 5 100 pairs of depth images and the number of objects lifted The bin picking simulation is shown in Fig 3 The training data is sampled such that the number of cases of picking a single object is the same as the number of cases of picking more than one object 90 of the data is used to train the CNN and the remaining 10 is used for validation of the trained CNN 7987 The training dataset was augmented by rotating and inverting the depth images the number of training data obtained at the end was 36 720 For each of the five kinds of objects a dataset was collected and then a respective CNN was trained using the corresponding training data After training we conducted picking experiment to test the CNN for 20 times for each object For each test the number of objects included in a bin is 35 for the objectA 27 for the object B 25 for the object C and 15 for the object D For the second experiment the goal was to evaluate the generalization capability of our method