增强现实中基于三维重建处理遮挡.doc

Neurocomputing () Contents lists available at ScienceDirectNeurocomputingjournal homepage: /locate/neucomHandling occlusions in augmented reality based on 3D reconstruction methodYuan Tian n,1, Yan Long, Dan Xia, Huang Yao, Jincheng ZhangSchool of Educational Information Technology, Central China Normal University, Wuhan 430079, ChinaPlease cite this article as: Y. Tian, et al., Handling occlusions in augmented reality based on 3D reconstruction method, Neurocomputing(2015), /10.1016/j.neucom.2014.12.081ia r t i c l e i n f o Article history:Received 16 October 2014Received in revised form25 December 2014Accepted 30 December 2014Communicated by Rongrong JiKeywords: Augmented reality Occlusion handling3D reconstructiona b s t r a c t The correct relationships between real and virtual objects are of utmost importance to a realistic augmented reality system, in which the occlusion handling method should be able to estimate the spatial relationships between real and virtual objects, as well as handle the mutual occlusion automatically in real-time. To accomplish the above tasks simultaneously, we propose a novel occlusion handling method based on 3D reconstruction, which consists of ofine stage and online stage. In the ofine stage, we get the depth map of the real scene using a low cost RGB-D camera. Then the 3D coordinate of each point in the global coordinate system are obtained and will be used in the online occlusion handling stage. In the online stage, we design a GPU based 3D point clouds alignment method by using point to tangent plane distance as error metric to accelerate the convergence speed and reduce the iterations. The correct relationships between real and virtual objects are then obtained automatically by comparing each pixels Z coordinate value of real objects with that of virtual objects in a smaller region to achieve real-time performance. More specically, we can judge and handle the mutual occlusion without human interactivity in real time, and experimental results prove its effectiveness.& 2015 Elsevier B.V. All rights reserved.1. IntroductionAugmented reality (AR) systems aim at adding virtual objects to real scene to make the virtual objects merge with the existing world seamlessly in such a manner as to appear part of the viewed3D scene 15. Applications include entertainment, military, education, machine manufacturing, and computer-aided surgery. Most nowadays systems simply overlay the virtual object onto the real video sequence and only try to minimum the registration errors 69, which is effective only when there is no occlusion between virtual objects and real scene.As shown in Fig. 1, a virtual object is added at the location of the marker in the real scene. As can be seen from Fig. 1(a), we can nd the spatial relationship between the virtual and real objects, if we look down from the top of the scene. The virtual object should be partially occluded by the real one (the red can) as shown in Fig. 1(c) when we observe in front of the scene. If we overlay the virtual object onto the scene regardless of the actual occlusion, we will get the result as Fig. 1(b) which will mislead the user to consider that the virtual object is in front of the red can. Moren Corresponding author.E-mail address: tianyuan_ (Y. Tian).1 Postal address: Room 626, Building 9, No.152 Luoyu Road, Wuhan 430079, Hubei, China.seriously, this may lead to eyestrain or motion sickness when people use it for a long time. In the “VIRTUE” project funded by German Research Foundation, researchers developed a medical AR system-ARGUS which can train the surgeons to perform surgery 10. However, in the actual training process, lacking of displaying the correct occlusion between the virtual and real objects greatly limits the users understanding, which has great impact on the systems usability. Therefore, a practical AR system should be capable of judging and exhibiting the occlusion between the virtual and real objects automatically.In this paper, we present a fully automatic system that judges the spatial relationships between real and virtual objects and handles mutual occlusion automatically in real time. We are interested in three aspects. First, in AR systems, there are many real objects and virtual objects occluding each other in the scene simultaneously. The previous contour based method 11 handles occlusions in two dimensions other than three dimensions, so it cannot get the correct occlusion relationships. To solve this problem, we reconstruct the dense 3D coordinates of the scene to handle mutual occlusion. Second, it is important to achieve the real-time performance. Researchers resort to various approaches to ensure that the registration processing speed is greater than 15 frames per second. The occlusion handling speed should also be fast enough to realize a practical AR system. In this paper, while we reconstruct the 3D model of the real scene in off-line stage, we/10.1016/j.neucom.2014.12.0810925-2312/& 2015 Elsevier B.V. All rights reserved.Fig. 1. Occlusion problem existing in AR systems. (For interpretation of the references to color in this gure legend, the reader is referred to the web version of this article.)only deal with the corresponding relationship between current frame and initial frame and then compare the Z coordinates of the real objects with that of the virtual objects in on-line stage. Therefore, the system runs in real time. Third, the system obtains the 3D coordinates of the real scene in reconstruction procedure in order to handle the occlusion problem automatically; whereas the previous research needs the users interactivity to get the contour of the occluding real object and it largely reduces the practicability of AR systems.The novelty in this paper is as follows: (1) we propose a noise reduction approach for the scenes from a RGB-D camera, (2) we use the distance from the point to the tangent plane as the error function and a GPU based iterative closet point algorithm to accelerate the convergence time and (3) we compare each pixels Z-coordinate of real objects with that of virtual objects in a small region covered by the re-projection to handle mutual occlusion.The remaining parts of this paper are organized as follows: Section 2 is related work and our contributions. Section 3 describes the system overview in detail. Sections 46 present the noise reduction approach, off-line 3D reconstruction method and on-line occlusion handling procedure separately. In Section 7, we give several experimental results to demonstrate the efciency of our method, and close with a conclusion in Section 8.2. Related work and our contributionsDue to the increasing interest of handling occlusions in AR applications and the implementation of displaying the correct spatial relationship of virtual and real scene, many methods have been proposed in the literature. Subsequent surveys of occlusion handling methods have been divided into three categoriescon- tour based method, depth based method and 3D reconstruction based method.2.1. Contour based methodsTian et al. 11 obtained the contour of the real object by using interactive segmentation method, and then tracked the object contour in the subsequent frames in real-time to display the correct occlusion relationship by redrawing all the pixels of the tracked object on the augmented image. It, however, needs users interactivity. Hence, Tian et al. 12 proposed an automatic occlu- sion handling method, extracting the contour of the occluding real object automatically through calculating the disparity map of the real scene in the rst frame, but the occlusion handling result is largely inuenced by the automatic contour exaction result in the rst frame. To reduce contradictory occlusions, Fukiage et al. 13 took advantage of characteristics of human transparency percep- tion without precise foreground-background segmentation, which is proved to be robust and real time even though there are complicated foreground objects in the scene. Sanches et al. 14segmented the real element in real-time and then performed OpenGL frame buffer operations to recover the pixels belonging to virtual objects to handle mutual occlusion, but there must be only one real object moving and many virtual objects based on ducial markers. Since the contour based method should estimate the occlusion before foreground-background segmentation, it will be ineffective when the occlusions change in the subsequent frames.2.2. Depth based methodsThese methods aim to obtain the correct occlusion by comparing each pixels depth value of real objects with that of virtual objects, and just display the un-occluded part of the virtual objects on the nal synthetic image as the virtual object is occluded by the real object. Schmidt et al. 15 introduced to compute dense disparity maps for a stereo image pair for detecting and handling occlusions in augmented reality. Hayashi et al. 16 detected moving objects from the dynamic background and then performed contour based stereo运动场景 提取运动轮廓获取深度 matching to get the depth of the region around the moving object in the real environment. However, when the real objects are static in the scene, the proposed method cannot present correct occlusion between virtual and real objects. Lu and Smith 17 developed a real-time occlusion handling method by segmenting objects and calculating depths of the areas covered by the virtual objects. To accelerate the processing speed, they useda GPU-based segmentation algorithm to segment the potentially occluded areas for matching. Their method is proved to be real-time and efcient when virtual objects and real objects move independently in the scene, but it cannot handle mutual occlusions. Zhu et al. 18 fused color, depth and neighbors to estimate depth and extract interested objects from scenes, and it is proved to be robust and efcient and be capable of handling mutual occlusions. Behzadan and Kamats method 19 can handle occlusions in dynamic AR environments in real-time by using depth sensing equipment and be easily integrated into any mobile AR platform. Arai et al. 20 employed epipolar geometry to solve occlusion problem in Intelligent Room, but their system required depth sensors measuring 3D information of the room and caused errors around borders of the objects. 13智能家居 ； 物体边缘出现错误Dong and Kamat 21 described a robust approach to correctly resolve visual occlusion in outdoor augmented reality by using real time time-of-ight (TOF) camera data and OpenGL frame buffer. Their method is efcient in both indoor and outdoor environments. However, the TOF camera and the video camera are not overlapped and the alignment of these two cameras is still a problem to be solved, which will cause the backgrounds captured by TOF camera are gray scale. To solve this problem, Dong et al. 22 used stereo projection method to nd the correspondence between the depth map and the RGB image captured by TOF camera and video camera separately. Although this approach can resolve AR occlusion in ubiquitous environments, the occlusion should be disabled when the TOF camera is moved around. Actually, depth based methods is time consuming and the depth information shouldY. Tian et al. / Neurocomputing () 9be re-calculated to obtain correct virtualreal occlusion relationship, especially, when the angle of view changes or the real scene changes.2.3. 3D reconstruction based methodsThese methods 2327 build the 3D model of the real scene and compare the depth of the virtual objects with that of the real scene to handle the occlusion. However, 3D reconstruction is time consuming especially for large and complicated scene. Fuhrmann et al. 23 simulate the occlusion of virtual objects by a represen- tation of the user modeled as kinematic chains of articulatedOffline StageObtain the depth map and RGB informationReduce noise3D reconstructionOnline StageObtain the depth map and RGB informationOcclusion handlingShow synthetic image with correct occlusionssolids. This method is limited to static scenes. Ong et al. 24 let the user outline the boundary of the occluding object on the key- views. Then the 3D occluding boundary was recovered from two consecutive key views. Finally the 2D occluding boundary was obtained by re-projecting the 3D occluding boundary in inter- mediate frames and the correct occlusion was realized. However,3D model will change in the intermediate frames due to the motion of the viewpoint which will result in inaccuracy of occlusion handling. Lepetit et al. 25 made some improvements. They used two consecutive key views to build the 3D occluding boundary and rened the boundary. But the drawback still exists. It would result in poor occlusion handling, especially when the viewpoint exceeds the range of key frames.Calculate the depth information of the real scene precisely in real time is the key of depth-based occlusion handling methods. But the calculation is complex and more than two cameras are always needed. These methods are suitable to static scenes and viewpoint. Although the computation cost of contour based methods is less, these methods cannot deal with mutual occlu- sions. 3D reconstruction based methods can deal with mutual occlusions and large viewing angle efciently.Our work is related to the 3D reconstruction based methods. In this paper, we present a practical framework for the occlusion handling method using a single RGB-D camera and we highlight our contributions below.1. We employ a low cost RGB-D camera equipped with a RGB camera, an infrared projector and an infrared sensor to obtain the depth map of the real scene, and we propose a noise reduction approach to reduce the noises in the depth map to obtain good performance and computation efciency simultaneously.2. We align the point clouds in different coordinate system to the same global coordinate system for tracking camera pose, but the traditional iterative closet point algorithm is of low itera- tion speed and easily falls into local minimum. Hence, we use the distance from the point to the tangent plane as the error function instead of point-to-point distance to accelerate the convergence time. Moreover, GPU based iterative closet point algorithm is utilized to further reduce the running time.3. In on-line stage, to reduce the computing time and achieve real-time performance, we design to compare each pixels Z- coordinate of real objects with that of virtual objects in a small region covered by the re-projection. This will solve the mutual occlusion problem automatically.3. System overviewAs can be seen from Fig. 2, our system is divided into two stages: ofine stage and online stage.Ofine stage mainly deals with the problem of noise reductionand 3D reconstruction. With the depth map and RGB information obtained from a RGB-D camera, we rst use the method intro- duced in Section 4 to complement some missing depth informa- tion caused by the nature of the camera hardware andFig. 2. Overview of the proposed system.environment. We then reconstruct the real scene using the approach presented in Section 5, which will directly inuence the effect of online occlusion handling results.In online stage, the correct occlusion relationships between real and virtual objects are estimated automatically by comparing the Z coordinates of the virtual objects with that of the real objects, which is introduced in Section 6 in detail. Subsequently, the synthetic image with correct occlusions is shown in real time.4. Noise reductionWe can obtain the 3D information of the real scene from many kinds of devices, such as 3D laser scanner, stereo camera, time of ight camera etc., which cost about $2-100k, a heavy economic burden for most researchers and consumers. Although the price of some time of ight cameras reduced to around $1k, their resolu- tion and frame rate is very low. When Microsofts Kinect is released, it attracts a lot of attention from researchers because of its low cost and excellent performance. Kinect is equipped with two sensors: a color camera and a depth camera. The infrared projector emits a non-uniform infrared pattern on the scene and the infrared sensor receives the same pattern to compute the depth map. But the depth map returned by Kinect is bound to be noisy due to three reasons:First, error caused by the Kinect device itself. It mainly refers to calibration error and measure error. The Kinect should be cali- brated, or it will inuence the accuracy of calculated depth map. Generally, this kind of error can be avoided if the calibration procedure is carried out properly, but the measuring error cannot be avoided. For instance, if we use Kinect to obtain the depth map of a white at wall, the depth values of the wall will be different