Applied-Research-of-Somatosensory-Game-Based-on-Kinect-and-Unity-3D-Data-Integration-Technology(1)

Applied Mechanics and Materials Vol 667 2014 pp 177 182Submitted 2014 08 22 2014 Trans Tech Publications SwitzerlandAccepted 2014 08 23 doi 10 4028 2014 10 08 Applied research of somatosensory game based on Kinect and Unity3D data integration technology Jingang Liu Computer Engineering Department Taiyuan Institute of Technology Taiyuan Shanxi Province China email jingang liu Keywords KinectUnity3Dsomatosensory game Abstract The paper analyzes data integration technology about Kinect and Unity3D Authors design scheme based on WPF and Unity 3D internal calling mode System includes screen show module Unity3D and Kinect interface module and data acquisition module There are scene settings rigging mirroring sports close range model smoothing processing and other functions in Unity3D and codes implement the device control rigging algorithm equipment image acquisition in Kinect Tested by C on unmanaged dll s management it is a good scheme to import Kinect hardware driver program and calling a custom data structures and algorithms to achieve the unity 3D scene In the unity 3D scene Kinect somatosensory camera control motion of models to improve the development efficiency of somatosensory game which has certain social value in development and application of somatosensory game Introduction For the past few years the proportion of the game has been more and more in popular entertainment market survey found the mobile games account for a large part secondly the online games and the stand alone games on computer However these games appear many disadvantages with producing much economic value and social value in these games most of the design need players become addicted and cost much time and money that effect players normal life and health so doped semi pros and cons of mobile game design and development is that many people are opposed to As technology advance it will be direction of future game development that human computer interaction like speech recognition gesture recognition closer to people s natural behavior When Microsoft released later on the Xbox Kinect2 0 somatosensory began quietly matured Microsoft s Kinect does not require the use of any controller and it relies on the camera to capture three dimensional movement of players but also to recognize human faces The game based on Kinect allows players to automatically connect recognizable voice accept commands 1 Currently in the somatosensory application software development mainstream development tools can be divided into two categories The first category is based on the two dimensional plane which is mainly based on Microsoft s WPF and WinForm second category is based on the three dimensional Developers are most familiar Unity3D game development tool that editors can run under Windows and Mac OS X platform the development of the game can be published to Windows Mac iPhone and Android platforms can also use the Unity web player plug in released web and mobile games support for web browsing 2 In Kinect applications and somatosensory game development process there is the development of inefficiencies therefore the main task is to make Kinect SDK and Unity 3D combine to control character models sport which reduce development costs in somatosensory game development All rights reserved No part of contents of this paper may be reproduced or transmitted in any form or by any means without the written permission of Trans Tech Publications ID 141 211 4 224 University of Michigan Library Media Union Library Ann Arbor USA 13 07 15 16 46 31 178Advances in Computers Electronics and Mechatronics The Key Technologies The design idea of Kinect is regarded human as the controller People appreciate the natural interaction shocking language powerfully However the high efficiency of Unity3D development module become the most popular game development engine But Unity3D this efficient development and somatosensory Kinect hardware is not compatible There are two kinds of Kinect SDK opening to developers interface mode One is based on C and the other is the way of the assembly which are not directly called by Unity3D 3 Fortunately the program provides unmanaged use in C Mono platforms and imports files whose filename extension is DLL Users can define the data structures and algorithms to reorganize data and be called in the Unity 3D WPF Kinect is Microsoft s XBOX360 somatosensory peripherals officially released the name in June 14 2010 which is the same as a 3D somatosensory camera with varies functions such as capturing motion image image recognition microphone input voice recognition community interaction etc Windows Presentation Foundation WPF is a next generation presentation system for building Windows client applications with visually stunning user experiences The core of WPF is a resolution independent and vector based rendering engine that is built to take advantage of modern graphics hardware WPF extends the core with a comprehensive set of application development features that include Extensible Application Markup Language XAML controls data binding layout 2 D and 3 D graphics animation styles templates documents media text and typography WPF is included in the Microsoft NET Framework so you can build applications that incorporate other elements of the NET Framework class library 4 The kernel of WPF The core of WPF is a resolution independent vector based rendering engine can take full advantage of modern graphics device WPF development and c integration can be realized through the Visual Studio 2008 application development Because of WPF and Common Language Runtime CLR fully integrated make full use of the CLR type safety cross platform features etc WPF s operating mechanism PresentationFramework and PresentationCore are based on the CLR open the way to managed code application programming interface API Milcore located under CLR is unmanaged code direct and DirectX tightly integrated which give user a good user experience Milcore is the core of the WPF hidden beneath the CLR drive components in the practical application development are not allowed to directly access Internal calling mode of Unity 3D Unity3D is a developed by Unity Technologies allows players to easily create such as 3D video game architectural visualization real time 3D animation and other types of interactive content multi platform integrated game development tool and its internal call structure is shown in Fig 1 system design and implementation System design is divided into Unity3D scene display module Unity3D interface module and Kinect data acquisition of three modules 5 Applied Mechanics and Materials Vol 667179 Program set Modules Class Delegate Types Enum Structure EventsFieldsAttributesMethods Fig 1 Unity 3D structure of internal callFig 2 Character model building effect Scene Module Scene modules can realize character model with body feeling camera to change Effect of model building is shown in Fig 2 Firstly putting a character in the scene secondly character modeling in the 3DMax thirdly skeleton binding in Unity3D binding at last adjusting the lighting and the sky Design of Unity3D interface module The Unity3D interface module for the attribute late Tim scripting and Unity3D the corresponding implementation rigging mirror movements close range model smoothing function 1 Skeletal binding In Kinect standing mode can track 20 points of articulation The NUI skeleton track is divided into active and passive two modes which provides up to two complete skeleton tracking data Under the active mode calling relevant frame and function to obtain the user s bone data under the passive mode supports up to four individual skeletal tracking but the location information of the user only contained in the mode not including the bone data in detail After body binding it is convenient to access data in the Unity3D scene 6 A array of bones present each bone node model which completes the mapping function of bone In the skeleton transform process four elements are used The four elements are a kind of simple hyper complex number The four elements are linear combinations composed of real number and i j kin Eq 1 and Eq 2 i2 j 2 k 2 1 1 a bi cj dk 0 2 Quaternion is used to express the object rotation in Unity3D After some calculation of rotation angle notably used four elements Quaternion The Quaternion variable is less let x y Z w on the 1 1 value Object rotation two weeks all its values of initial value initial value 0 0 0 1 if along the Y axis rotation of 180 360 540 720 respectively corresponding angle value 0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 if the the X axis rotation of 180 360 540 720 respectively corresponding angle value 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 No rotation is written with Quaternion identify 180Advances in Computers Electronics and Mechatronics 2 Mirror Sports Mirror Sports is one kind of the limb movement for voluntary movement the other limbs appeared almost simultaneously copied involuntary movement motion model is consistent with the actual movement pattern mirrored after the binding character if not match and it will move in the opposite direction When creating a replacement in the bone and joint user specifies the joint index number to edit different joints When in skeletal mapping the joint index corresponding to the dynamic change then the variables determine Setting the Boolean variable Mirrored Movement using the ternary operator returns the index number MirroredMovement 8 12 Different indexes reflect the different parts 3 Close up mode In Kinect for Windows SDK 1 0 and above versions under the close up mode Kinect can sit or 10 joints mode skeletal tracking tracking only the upper body of the head neck shoulders arms and other 10 joints When the human skeleton binding the binding station and ride a few different joints 7 Define Data Structure as follows public enum NuiImageStreamFlags None 0 x00000000 SupressNoFrameData 0 x0001000 EnableNearMode 0 x00020000 TooFarIsNonZero 0 x0004000 Display the calling statement in the API as follows if isNearMode KinectWrapper NuiImageStreamSetImageFrameFlags depthStreamHandle NuiImageStreamFlags EnableNearMode The NuiImageStreamSetImageFrameFlags function is used to complete the type selection of the data frame got just now when choose the close up mode ready to start the closing up mode identifying only the joints under the closing up mode so that the tester sits in front of the camera to achieve the binding 4 Smoothing Handle Smoothing makes the bones rotate freely in case the jumping and moving in a second efficiency it can be achieved by the Bone Rotation Algorithm First declaring the variable TransformSmoothParameters and set the values to the paraments of the Smoothing Algorithm the method is as follows TransformSmoothParameters smoothParameters smoothParameters Smoothing 0 5f Set the amount of smoothing of the skeleton data frame receiving a floating point value from 0 to 1 the larger the value the more smooth 0 means no smoothing Next handle it by smoothing method the method is as follows boneTransform rotation Quaternion Slerp boneTransform rotation newRotation Time deltaTime SmoothFactor Kinect data reading module Kinect data reading module contains two parts device control and Skeletal bonding 1 Device Control The device control is mainly reflected in the boot device and initialization the camera angle adjustment turn off the device Applied Mechanics and Materials Vol 667181 Because the Unity3D internal does not support the Microsoft s SDK so we import Kinect hardware drivers the code to control the device as follows DllImportAttribute Kinect10 dll EntryPoint NuiInitialize public static extern int NuiInitialize NuiInitializeFlags dwFlags DllImportAttribute Kinect10 dll EntryPoint NuiShutdown public static extern void NuiShutdown DllImportAttribute Kinect10 dll EntryPoint NuiCameraElevationSetAngle public static extern int NuiCameraSetAngle long angle 2 Skeletal binding Bonding the joints s methods are joint bonding control get the next frame of the joint bones smooth mapping bone rotation calculations The code with the joint bonding is as follows DllImportAttribute Kinect10 dll EntryPoint NuiSkeletonTrackingEnable public static extern int NuiSkeletonTrackingEnable IntPtr hNextFrameEvent uint dwFlags DllImportAttribute Kinect10 dll EntryPoint NuiSkeletonGetNextFrame public static extern int NuiSkeletonGetNextFrame uint dwMillisecondsToWait ref NuiSkeletonFrame pSkeletonFrame DllImportAttribute Kinect10 dll EntryPoint NuiTransformSmooth public static extern int NuiTransformSmooth ref NuiSkeletonFrame pSkeletonFrame ref NuiTransformSmoothParameters pSmoothingParams DllImport Kinect10 dll EntryPoint NuiSkeletonCalculateBoneOrientations public static extern int NuiSkeletonCalculateBoneOrientations ref NuiSkeletonData pSkeletonData NuiSkeletonBoneOrientationpBoneOrientations Test Scene display interface modules data acquisition modules are tested from the data consistency debug error limit test results are given in three areas as listed in Table 1 and Table 2 Table 1 Module test results ModuleConsistency of the dataDebug displayError limits Scenario showsData transmissionnone0 Interface ModuleSkeletal bindingnormal0 1 Data AcquisitionColor ImageMap Bones 1 Table2 Gesture recognition parameter values Gesture TypeTesting timesCorrect numbersRecognition rate Wave1500149099 2 ClicConclusion In the design scene all the functions of data reading skeletal binding and image capturing are completed Inter module calls properly data are displayed correctly and transmission stability Kinect SDK has built in gesture recognition function I