IROS2019国际学术会议论文集0687

Multi-Hand Direct Manipulation of Complex Constrained Virtual Objects Jun-Sik Kim, MyungHwan Jeon and Jung-Min Park AbstractWe present a method to manipulate virtual objects which are constrained complexly and reconfi gurable as if they would exist in a real world by using multiple hands simultaneously. A complexly constrained, and reconfi gurable object, such as a Rubiks cube, is hard to describe its physical motion constraints, mainly because they are determined by the grasping situation and dynamically changeable. Rather than describing the physical motion constraints in a general form, we more focus on the multiple hand interaction of a complex object. A complex object is divided into multiple subparts which are grasped by each hand, and the constraints between the subparts are optimized for inducing natural and continuous movement. For this, we propose a dynamically adjustable data structure for representing object parts grasped by multiple hands, and an optimization-based pose estimation of the constrained subparts along with their grasped hands. The experiments show that human subjects can manipulate a complexly constrained object such as a Rubiks cube without any diffi culty as if it exists in the real-world. I. INTRODUCTION Many devices for virtual reality (VR) and augmented reality (AR) have been distributed in the market, and have become one of the major platforms for gaming including head-mounted displays and smartphones. Though they have been driven many interesting applications of VR/AR, their user interactions are still indirect: using hand-held devices for raycasting or predefi ned gestures. 1, 2 There is no direct user interface which allows users to touch and manipulate virtual objects as if they do in their real life. Direct and natural user interface will make users feel more immersion in the VR/AR applications, but there are many remaining challenges. Aside from the lack of proper haptic feedback devices, manipulation itself has diffi culties to implement it properly. There exist countless number of ways in manipulating and using a real world object, and there are many combinations of objects. To induce their responses against a users action, there are two possible approaches. First one is to defi ne possible responses for each object. In this approach, an object does a predefi ned response, and thus, interaction methods become limited. This work was supported by the Global Frontier R&D Program on “Human-centered Interaction for Coexistence“ funded by the Na- tional Research Foundation of Korea grant funded by the Korean Government(MSIP)(2011-0031425) Jun-Sik Kim and Jung-Min Park are with the Center for Intelli- gent and Interactive Robotics, Korea Institute of Science and Tech- nology,Hwarangro14gil5,Seongbuk-Gu,Seoul02792,Korea. junsik.kimkist.re.kr.pjmkist.re.kr MyungHwan Jeon is a master student in the Robotics Program, KAIST, Korea. This work has been done when he was with the Center for Human- centered Interaction for Coexistence, Korea.jmong1994 (a) Grasping and handover (b) Manipulating a subpart (c) Interacting with another object (d) Manipulating a hinge-constrained object Fig. 1: Various types of two-hand interaction with a various type of virtual objects including a Rubiks cube. The other way is to utilize a physics simulator for gen- erating responses of an object. In fact, the idea using a physics simulator for object manipulation is not new, rather classical in robotics society. Object manipulation by hands has been one of the major investigation topics for robotic grasping. One traditional way is to use shape primitives as a prior knowledge to plan the object grasping. 3, 4, 5 This approach can not be applicable to the VR/AR interaction, because it usually uses a simple hand model for a robotic gripper, but direct manipulation by a human hand is more complicated. Another track of research is data-driven: collecting huge number of grasping data and generating possible grasping plan based on the dataset 6, 7, 8, 9 This approach is more fl exible and applicable to a real human hand, but most of the work is only for grasping planning, not for sequential interaction. Realistic direct manipulation has been investigated through physics simulation in VR interaction. One representative work has been done by Kumar and Todorov 10, who devel- oped a physics engine MuJoCo to induce a hand interaction with virtual objects. In this work, dynamics of all the objects and hands are solved through a single physics simulation so 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Macau, China, November 4-8, 2019 978-1-7281-4003-2/19/$31.00 2019 IEEE3235 (a) Rubiks Cube(b) Left grasping and the constraint axis(c) Top grasping and the constraint axis Fig. 2: Examples of a dynamically reconfi gurable object, a Rubiks Cube. Depending on the grasp position and direction, the constraint axis should be dynamically set. that the objects responds as they follow a law of physics. Hilliges et al. 11 developed a particle based hand inter- action using a conventional physics simulator which causes realistic response of virtual objects. Kim and Park 12 made a combination of deformable meshes and physics particles in order to track the particles fast enough to interact with virtual objects as well. It has been extended into multi-hand interaction and manipulating simply constrained objects 13, 14 by optimizing minimum number of required variables describing the virtual objects. However, there exist much more complicated objects in the real world. Many objects are combinations of the other objects, and sometimes, the combination structure changes. Fig. 1 shows various types of multi-hand interaction for constrained objects to each other. Especially, a Rubiks cube is interesting. As shown in Fig. 1 (a) and (c), the whole set is just a rigid body object. However, the internal structure can be easily modifi ed as shown in Fig. 1 (b). The cube elements are tightly coupled to each other, but also their confi guration can change dynamically. It is a single object, but also a set of objects. It is not trivial to describe this complicated and dynamic structure, and to induce its response correctly. Fig. 2 shows challenges in describing the object response. As one can see in Fig. 2 (b) and (c), possible internal rotation is determined by how a user grasps it. By grasping its side, the constraint is set to a single axis rotation through the X- axis. By grasping its top, it is set to a rotation through the Z-axis, even though the objects in the group do not change at all. In addition, for grasping a Rubiks cube, the fi ngers are across multiple small cubes. In the real world, the small cubes are tightly attached, but in VR situation, each object is independently defi ned. Thus, the grasping a multi-body object is not also trivial. In this paper, we extend the work of Kim and Park 14 so that it can deal with those dynamically reconfi gurable objects. We propose a hierarchical tree structure for de- scribing composition of the objects consisting multiple small objects. The proposed object representation is general, and it is possible to defi ne a single rigid body and a simply constrained object as well. II. OBJECT-CENTEREDINTERACTION ANDOBJECT REPRESENTATION The ultimate goal of this work is to make virtual object manipulation as realistic and natural as possible. The realistic Grasping Grouped Object Subgroup 1 1823 . Grouped Object Subgroup 1 Subgroup 2 17234568 hand1 hand2 R,t Fig. 3: Object representation of a dynamically reconfi gurable object based on hand interaction. and natural response of the virtual object implies that the pose or confi guration of the object changes seamlessly, without any abrupt change. For a constrained object, Kim and Park 14 noticed that formulating more degree-of-freedom of the object status than actually required induces its abrupt change, and proposed to use the minimum degree-of-freedom formulation in optimizing its status including its pose and constrained variables. We extend their work for more general and complexly constrained objects such as a Rubiks cube by dynamic grouping of objects related to grasping situation. A. Grasping-based Object Representation We redefi ne the object representation based on interaction between hands and the objects. The proposed object repre- sentation is a dynamic tree structure representing hierarchical parent-child relations. Fig. 3 illustrates the proposed object representation based on hand interaction. Each object has at least one subgroup that contains primitive objects or other subgroups as its children. A subgroup is a tree node representing a local coordinate system and the relative transformation of its every child from the node is fi xed. Each subgroup can be considered as a rigid-body object in the manipulation, and at the fi rst time, all the elememts are the children of the only subgroup as shown in the left fi gure in Fig .3. Once the object is grasped with one hand as shown in Fig. 2 (b) or (c), the grasped elements build a subgroup depending on the object manipulability. The right fi gure in Fig. 3 shows an example containing two subgroups which are constrained a single axis rotation , which is an example of a 2 2 2 Rubiks cube as shown in Fig. 2. If another hand grasps the element objects in subgroup 2, the 3236 Hand 1 Hand 2 Group 1 Group 2 Fig. 4: Two-hand manipulation of a Rubiks cube. single axis rotation can be estimated by the hand motion. When the subgroups are set, the corresponding constraint is determined, and the optimization using the minimum DOF representation 14 is utilized for each grouped object, as if there are two rigid bodies constrained to each other. Note that in the representation diagram, there are solid arrows which represent fi xed transformations, and the dashed arrows which are adjustable variables in. Because generating sub- groups are highly related to the hand grasping, we call this representation grasping-based object representation. Each subgroup potentially implies a one-hand grasping part among the elements of the grouped object. Once the grasping is released, the hierarchical tree is set back to the left one in Fig. 3, by simply merging the two subgroup. We should mention that this structural property makes handover simple: by just building another subgroup for the second hand and detaching the fi rst one, the handover between two hands is accomplished. B. Object-specifi c Grasping Criteria - Case of Rubiks Cube Each object or grouped object should be independently defi ned, and for its defi nition, it requires 1) to describe ele- ment objects and their initial transformations, 2) to determine grasping criteria for establishing a subgroup, and 3) to defi ne the proper constraints based on the generated subgroup. Description of the element objects is simply adding shapes to the tree along with their transformation. But the grasping criteria is depending on the object constraints and its shapes. For a simple rigid-body object, checking the direction of the penetrating physics particles for the object is good enough. After collecting contacted particles of a hand, the following criteria is used 12 d i dj (1) where i and j are the indices of the contact particles and, diand djare the corresponding penetrating directions. The threshold is set as 0.7 usually. However, making a group implies that the multiple objects are grasped at the same time with a single hand. As shown in Fig. 4, each small block is not grasped by a hand, but subgroups as rigid bodies are grasped. Thus, the grasping criteria should be redefi ned as a property of a grouped object. One can notice that every combination of the eight blocks can not make a proper subgroup: only blocks on the same Fig. 5: Cooperative manipulation by remote users on 3D display. side can make a valid subgroup. To check the validity, we make a canonical coordinate system for a Rubiks Cube. In any confi guration, the relative positions of the element cubes are retrieved in the canonical coordinate in order to check the blocks in each side as fast as possible. For checking the grasping, every possible side blocks are assumed as a rigid body and check the criteria (1) as long as at least three blocks are contacted on the group to assess possible ambiguity. Once the subgroup is defi ned in the canonical coordinate, the constraint axis shown in Fig. 2 is trivially determined as a center axis of the subgroup in the canonical coordinate system. This grasping criteria and the defi nition of the constraint are object dependent and should be defi ned for each grouped- object. For example, for a simple hinge object, simply using the method (1) and using predefi ned hinge property is enough. C. Relation between Grasping Hand and Subgroup Each subgroup is controlled by its grasping hand: the subgroup as a rigid body object has a fi xed relative transfor- mation to its grasping hand, so that the subgroup elements seems to be attached to the hand. However, the hand motion can not be physically constrained to be matched the physical confi guration of the object. For example, Fig. 4 shows that two hands grasp a single Rubiks cube, and there should be two subgroups for each hand which can freely move. The primary subgroup, i.e. the fi rst generated subgroup, is related to the fi rst grasping hand, and its pose is controlling the whole object. The pose of the other subgroups is opti- mized, and its relative transformation from the corresponding grasping hand is changing. Because the non-primary sub- group pose is updated so that the shape of the grouped object is well-defi ned after the optimization with the minimal DoF, the relative transformation T0relof a subgroup to the grasping hand should be updated as T0rel= T0objT1 hand (2) where T0objis a new object pose which is updated by the optimization, and Thandis the pose of the grasping hand. D. Cooperative Manipulation by remote users Because the proposed method distinguish neither the owner of hands nor their directions, multi-user interaction 3237 User Leap Motion Sensor 2D Display Fig. 6: Experimental setup. For motion capture of hands and fi ngers, a Leap Motion sensor has been used. is simply achieved by registering the remote hands to the physics world. Fig. 5 shows the example of the multi-user coorperative manipulation, that is, one user holds the cube and the other rotates the part of the cube on immersive 3D display systems. To do this, each agent reads and reports its hand status to the server, and the server system computes the updated poses of objects, and sends the pose information to all the agents to update the scene. When the server updates the pose of the constrained object, it sends the poses of each element object separately back as if there are multiple independent objects. This works because the non- server agent does not run any physics simulation, and the hierarchical data structure is not required. III. EXPERIMENTS To validate the proposed multi-hand interaction method, we made experiments manipulating a Rubiks cube object. We asked human subjects to grasp and manipulate the object as indicated, such as “grasp the object in X direction with the left hand, and rotate the other side clockwise”. The success rate and manipulation accuracy have been analyzed in some controlling conditions. A. System Setup and Experimental Tasks Fig. 6 shows the test setup. For detecting hand motion, we used the Leap Motion Sensor. The used system was with Intel Core i7-3770 (3.5GHz) and 16 GB memory. All the system has been implemented on Openframeworks in C+ on OpenGL, and Bullet Physics 2.82 for physics simulation. 16 The whole process is running only on the CPU multithreaded by Boost 1.57 and OpenMP. The overall throughput including rendering has been measured up to 30 Hz, higher than sensor and video rate. Fig. 7 illustrates experimental tasks, from the top-left to bottom right. Once the task starts, the task instruction appears on the screen. The instruction includes 1) the primary grasping hand, 2) direction of the rotating axis, and 3) the rotating direction as clockwise and counterclockwise. A subject starts to manipulate the object as instructed after the 2 2 2 Rubiks Cube dropped randomly. Rotation angle was set to 90 degrees in all the trials. After every trial, human observer judged if the task was successful or not, with the criterion if the object behavior was not natural such as Fig. 7: Task specifi cation for experiments. abrupt changes, or the subject had diffi culties to manipulate the object as instructed. The system measures the time for manipulation, starting from the fi rst contact of one fi nger to the end of the task. B. Results and Analysis We compare the effect of the primary grasping hand and the rotating direction for manipulation accuracy and execution time. After that, we compare the manipulation performance of new users to that of a highly experienced user. For the experiments, eight subjects who use the system at the fi rst time are asked to do the given task. Their ages spread from early 20 to early 30, and six males and two females are included. 1) Success Rate and Manipulation Accuracy: Table I shows the task success rate and the error statistics of the cube manipulation. The success rate is over 95% for those who do not have any experience on the bare-hand manipulation system. In addition, the error statstics shows that the subject can make the task successfully in terms of the accuracy. Fig. 8 shows the error distributions in different manipula- tion situations. The selection of the primary hand does not affect the manipulation accuracy signifi cantly. However, one can notice that accuracy distributions are a bit wider in cases of the clockwise rotation while the left hand grasping and the counter-clockwise rotation while