Applied-Research-of-Somatosensory-Game-Based-on-Kinect-and-Unity-3D-Data-Integration-Technology(1).doc

Applied Mechanics and Materials Vol. 667 (2014) pp 177-182Submitted: 2014-08-22 (2014) Trans Tech Publications, SwitzerlandAccepted: 2014-08-23doi:10.4028//AMM.667.177Online: 2014-10-08Applied research of somatosensory game based on Kinect and Unity3D data integration technologyJingang LiuComputer Engineering Department, Taiyuan Institute of Technology, Taiyuan Shanxi ProvinceChinaemail: jingang_Keywords: KinectUnity3Dsomatosensory gameAbstractThe paper analyzes data integration technology about Kinect and Unity3D. Authors design scheme based on WPF and Unity 3D internal calling mode. System includes screen show module, Unity3D and Kinect interface module and data acquisition module. There are scene settings, rigging, mirroring sports, close-range model, smoothing processing and other functions in Unity3D; and codes implement the device control, rigging algorithm, equipment image acquisition in Kinect. Tested by C # on unmanaged dlls management, it is a good scheme to import Kinect hardware driver program and calling a custom data structures and algorithms to achieve the unity 3D scene. In the unity 3D scene, Kinect somatosensory camera control motion of models to improve the development efficiency of somatosensory game, which has certain social value in development and application of somatosensory game.IntroductionFor the past few years, the proportion of the game has been more and more in popular entertainment, market survey found, the mobile games account for a large part, secondly the online games and the stand-alone games on computer. However, these games appear many disadvantages with producing much economic value and social value, in these games, most of the design need players become addicted, and cost much time and money, that effect players normal life and health, so doped semi-pros and cons of mobile game design and development is that many people are opposed to. As technology advance, it will be direction of future game development that human-computer interaction, like speech recognition, gesture recognition, closer to peoples natural behavior. When Microsoft released later on the Xbox Kinect2.0, somatosensory began quietly matured. Microsofts Kinect does not require the use of any controller, and it relies on the camera to capture three-dimensional movement of players, but also to recognize human faces. The game based on Kinect allows players to automatically connect, recognizable voice, accept commands1.Currently, in the somatosensory application software development, mainstream development tools can be divided into two categories. The first category is based on the two-dimensional plane, which is mainly based on Microsofts WPF and WinForm; second category is based on the three-dimensional. Developers are most familiar Unity3D game development tool that editors can run under Windows and Mac OS X platform, the development of the game can be published to Windows, Mac, iPhone and Android platforms, can also use the Unity web player plug-in released web and mobile games, support for web browsing2.In Kinect applications and somatosensory game development process, there is the development of inefficiencies, therefore, the main task is to make Kinect SDK and Unity 3D combine, to control character models sport, which reduce development costs in somatosensory game development.All rights reserved. No part of contents of this paper may be reproduced or transmitted in any form or by any means without the written permission of Trans Tech Publications, . (ID: 24, University of Michigan Library, Media Union Library, Ann Arbor, USA-13/07/15,16:46:31)178Advances in Computers, Electronics and MechatronicsThe Key TechnologiesThe design idea of Kinect is regarded human as the controller. People appreciate the natural interaction shocking language powerfully. However, the high efficiency of Unity3D development module, become the most popular game development engine. But Unity3D this efficient development and somatosensory Kinect hardware is not compatible. There are two kinds of Kinect SDK opening to developers interface mode. One is based on C+, and the other is the way of the assembly, which are not directly called by Unity3D3. Fortunately, the program provides unmanaged use in C # Mono platforms, and imports files, whose filename extension is DLL. Users can define the data structures and algorithms, to reorganize data and be called in the Unity 3D.WPFKinect is Microsofts XBOX360 somatosensory peripherals officially released the name in June 14, 2010, which is the same as a 3D somatosensory camera with varies functions such as, capturing motion image , image recognition, microphone input, voice recognition, community interaction etc. Windows Presentation Foundation (WPF) is a next-generation presentation system for building Windows client applications with visually stunning user experiences. The core of WPF is a resolution-independent and vector-based rendering engine that is built to take advantage of modern graphics hardware. WPF extends the core with a comprehensive set of application-development features that include Extensible Application Markup Language (XAML), controls, data binding, layout, 2-D and 3-D graphics, animation, styles, templates, documents, media, text, and typography. WPF is included in the Microsoft .NET Framework, so you can build applications that incorporate other elements of the .NET Framework class library4.The kernel of WPFThe core of WPF is a resolution independent vector-based rendering engine, can take full advantage of modern graphics device, WPF development and c # integration can be realized through the Visual Studio 2008 application development. Because of WPF and Common Language Runtime (CLR) fully integrated, make full use of the CLR type safety, cross-platform features, etc. WPFs operating mechanism, PresentationFramework and PresentationCore are based on the CLR, open the way to managed code application programming interface (API). Milcore located under CLR, is unmanaged code, direct and DirectX tightly integrated, which, give user a good user experience. Milcore is the core of the WPF hidden beneath the CLR drive components, in the practical application development are not allowed to directly access.Internal calling mode of Unity 3DUnity3D is a developed by Unity Technologies allows players to easily create such as 3D video game, architectural visualization, real-time 3D animation and other types of interactive content multi-platform integrated game development tool, and its internal call structure is shown in Fig.1.system design and implementationSystem design is divided into Unity3D scene display module, Unity3D interface module and Kinect data acquisition of three modules5.Applied Mechanics and Materials Vol. 667179Program setModulesClass, Delegate,TypesEnum, StructureEventsFieldsAttributesMethodsFig.1 Unity 3D structure of internal callFig.2 Character model building effectScene ModuleScene modules can realize character model with body feeling camera to change. Effect of model building is shown in Fig 2. Firstly, putting a character in the scene, secondly, character modeling in the 3DMax, thirdly, skeleton binding in Unity3D binding, at last, adjusting the lighting and the sky.Design of Unity3D interface module:The Unity3D interface module for the attribute late Tim scripting and Unity3D the corresponding implementation, rigging, mirror movements, close range model, smoothing function.(1) Skeletal bindingIn Kinect, standing mode can track 20 points of articulation. The NUI skeleton track is divided into active and passive two modes, which provides up to two complete skeleton tracking data. Under the active mode, calling relevant frame and function to obtain the users bone data; under the passive mode, supports up to four individual skeletal tracking, but the location information of the user only contained in the mode, not including the bone data in detail.After body binding, it is convenient to access data in the Unity3D scene 6. A array of bones present each bone node model, which completes the mapping function of bone. In the skeleton transform process, four elements are used. The four elements are a kind of simple hyper complexnumber. The four elements are linear combinations composed of real number and i ,j , kinEq.1 and Eq.2.i2 = j 2 = k 2 = 1(1)a + bi + cj + dk = 0(2)Quaternion is used to express the object rotation in Unity3D. After some calculation of rotation angle, notably, used four elements Quaternion. The Quaternion variable is less, let x, y, Z, w on the -1,1 value. Object rotation two weeks all its values of initial value, initial value: (0,0,0,1) if, along the Y axis rotation of 180 , 360 , 540 , 720 , respectively corresponding angle value (0,1,0,0), (0,0,0, -1), (0, -1,0,0), (0,0,0,1); if the the X axis rotation of 180 , 360 , 540 , 720 , respectively corresponding angle value (-1,0,0,0), (0,0,0, -1), (1,0,0,0), (0,0,0,1). No rotation is written with Quaternion.identify.180Advances in Computers, Electronics and Mechatronics(2) Mirror SportsMirror Sports is one kind of the limb movement for voluntary movement, the other limbs appeared almost simultaneously copied involuntary movement, motion model is consistent with the actual movement pattern mirrored after the binding character, if not match, and it will move in the opposite direction.When creating a replacement in the bone and joint, user specifies the joint index number to edit different joints.When in skeletal mapping, the joint index corresponding to the dynamic change, then the variables determine. Setting the Boolean variable Mirrored Movement, using the ternary operator returns the index number. (! MirroredMovement ? 8 : 12) Different indexes reflect the different parts.(3) Close-up modeIn Kinect for Windows SDK 1.0 and above versions, under the close-up mode, Kinect can sit or 10 joints mode skeletal tracking, tracking only the upper body of the head, neck, shoulders, arms and other 10 joints. When the human skeleton binding, the binding station and ride a few different joints7.Define”Data Structure”as follows: public enum NuiImageStreamFlagsNone = 0x00000000, SupressNoFrameData = 0x0001000, EnableNearMode = 0x00020000,TooFarIsNonZero = 0x0004000;Display the calling statement in the API as follows: if(isNearMode)KinectWrapper.NuiImageStreamSetImageFrameFlags(depthStreamHandle,NuiImageStreamFlags.EnableNearMode);The NuiImageStreamSetImageFrameFlags( ) function is used to complete the type selection of the data frame got just now, when choose the close-up mode, ready to start the closing-up mode, identifying only the joints under the closing-up mode, so that the tester sits in front of the camera to achieve the binding.(4) Smoothing HandleSmoothing makes the bones rotate freely, in case the jumping and moving in a second efficiency, it can be achieved by the Bone Rotation Algorithm.First, declaring the variable TransformSmoothParameters, and set the values to the paraments of the Smoothing Algorithm, the method is as follows:TransformSmoothParameters smoothParameters;smoothParameters.Smoothing = 0.5f;Set the amount of smoothing of the skeleton data frame, receiving a floating point value from 0 to 1, the larger the value, the more smooth. 0 means no smoothing.Next, handle it by smoothing method, the method is as follows: boneTransform.rotation= Quaternion.Slerp (boneTransform.rotation, newRotation, Time.deltaTime * SmoothFactor);Kinect data reading moduleKinect data reading module contains two parts: device control and Skeletal bonding.(1) Device ControlThe device control is mainly reflected in: the boot device and initialization, the camera angle adjustment, turn off the device.Applied Mechanics and Materials Vol. 667181Because the Unity3D internal does not support the Microsofts SDK, so we import Kinect hardware drivers, the code to control the device as follows:DllImportAttribute(Kinect10.dll, EntryPoint = NuiInitialize) public static extern int NuiInitialize(NuiInitializeFlags dwFlags); DllImportAttribute(Kinect10.dll, EntryPoint = NuiShutdown) public static extern void NuiShutdown();DllImportAttribute(Kinect10.dll, EntryPoint = NuiCameraElevationSetAngle) public static extern int NuiCameraSetAngle(long angle);(2) Skeletal bindingBonding the jointss methods are: joint bonding control, get the next frame of the joint, bones smooth mapping, bone rotation calculations. The code with the joint bonding is as follows:DllImportAttribute(Kinect10.dll, EntryPoint = NuiSkeletonTrackingEnable)public static extern int NuiSkeletonTrackingEnable(IntPtr hNextFrameEvent, uint dwFlags); DllImportAttribute(Kinect10.dll, EntryPoint = NuiSkeletonGetNextFrame)public static extern int NuiSkeletonGetNextFrame(uint dwMillisecondsToWait, ref NuiSkeletonFrame pSkeletonFrame);DllImportAttribute(Kinect10.dll, EntryPoint = NuiTransformSmooth)public static extern int NuiTransformSmooth(ref NuiSkeletonFrame pSkeletonFrame,ref NuiTransformSmoothParameterspSmoothingParams);DllImport(Kinect10.dll, EntryPoint = NuiSkeletonCalculateBoneOrientations)public static extern int NuiSkeletonCalculateBoneOrientations(ref NuiSkeletonData pSkeletonData, NuiSkeletonBoneOrientationpBoneOrientations);TestScene display, interface modules, data acquisition modules are tested from the data consistency, debug, error limit test results are given in three areas, as listed in Table 1 and Table 2.Table 1 Module test resultsModuleConsistency of the dataDebug displayError limitsScenario showsData transmissionnone0%Interface ModuleSkeletal bindingnormal0.1%Data AcquisitionColor ImageMap Bones()1%Table2 Gesture recognition parameter valuesGesture TypeTesting timesCorrect numbersRecognition rateWave1500149099.2%ClicConclusionIn the design scene, all the functions of data reading, skeletal binding, and image capturing are completed. Inter-module calls properly, data are displayed correctly and transmission stability Kinect SDK has built-in gesture recognition function. In the design, custom defines gesture recognition with wave and click. In the test, data tran