IROS2019国际学术会议论文集 2323

3D Canonical Pose Estimation and Abnormal Gait Recognition with a Single RGB D Camera Yao Guo Member IEEE Fani Deligianni Xiao Gu Guang Zhong Yang Fellow IEEE Abstract Assistive robots play an important role in improv ing the quality of life of patients at home Among all the monitoring tasks gait disorders are prevalent in elderly and people with neurological conditions which increases the risk of fall Therefore the development of mobile systems for gait monitoring at home in normal living conditions is important Here we present a mobile system that is able to track humans and analyze their gait in canonical coordinates based on a single RGB D camera Firstly view invariant 3D lower limb pose estimation is achieved by fusing information from depth images along with 2D joints derived in RGB images Next both the 6D camera pose and the 3D lower limb skeleton are real time tracked in a canonical coordinate system based on Simultaneously Localization and Mapping SLAM A mask based strategy is exploited to improve the re localization of the SLAM in dynamic environments Abnormal gait is detected by using the Support Vector Machine SVM and the Bidirectional Long Short Term Memory BiLSTM network with respect to a set of extracted gait features To evaluate the robustness of the system we collected multi camera ground truth data from sixteen healthy volunteers performing six gait patterns that mimic common gait abnormalities The experiment results demonstrate that our proposed system can achieve good lower limb pose estimation and superior recognition accuracy com pared to previous abnormal gait detection methods I INTRODUCTION Gait disorders usually result from neurological or mus culoskeletal conditions and they are common in the elderly 1 2 In neurological diseases such as Parkinson s they mark the disease progression and severity In elderly they are associated with a high risk of falls poor quality of life and increased risk for depression Therefore there is a pressing need for real time gait analysis in patients homes 3 In the past decades various gait analysis systems have been developed for pathological gait detection 4 These include multi camera motion capture system multiple In ertial Measurement Units IMU system force plates and pressure insoles Although these systems can monitor the kinematics and dynamics of the lower limbs movements with promising precision they require participants to wear specialized markers sensors and involved complicated setup which limits their application in specialized hospitals or rehabilitation centers Recent advances in computer vision have demonstrated good performance in offl ine markerless gait analysis 5 This work was supported by Engineering and Physical Sciences Research Council EPSRC under Grant EP R026092 1 Y Guo F Deligianni X Gu and G Z Yang are with the Ham lyn Centre South Kensington Campus Imperial College London Lon don SW7 2AZ United Kingdom yao guo fani deligianni xiao gu17 g z yang imperial ac uk G Z Yang is also with the Institute of Medical Robotics Shanghai Jiao Tong University China 7 Furthermore real time 2D full body pose estimation from RGB images has been achieved 8 However most of the pathological gait involves atypical joint kinematics and dynamics The evaluation of the gait parameters in 2D space impedes objective clinical evaluation since the relation to standard clinical indices is ill defi ned 9 On the other hand depth sensors technology such as Kinect enables real time 3D human skeleton tracking 10 11 Supported by recent advances in robot vision and artifi cial intelligence it is possible to monitor a patient s health while carrying out daily activities 12 14 To this end a number of challenges should be addressed in order for robots to be aware of their surroundings follow humans and track their gait accurately Firstly gait analysis based on depth images ignores the abundant texture features in the color space and sometimes results in the unsatisfactory estimation of the lower limb joints Secondly the lower limb movement detection accuracy may vary with respect to the distance between the camera and the person while the human is observed from a fi xed perspective Thirdly the 3D lower limb pose is represented in the moving camera frame which means the joint trajectories and many signifi cant gait parameters e g the gait speed and the step length cannot be estimated without prior knowledge of the moving camera position To address the aforementioned challenges this paper presents an RGB D based mobile 3D gait analysis system for tracking both the 6D camera and the 3D lower limb pose in a canonical coordinate system The 3D map of the environment is pre built and stored offl ine The 3D lower limb pose is estimated in the camera frame coordinate system by fusing state of the art RGB based 2D pose estimation 8 with the depth inputs Subsequently 3D human pose estimation is mapped to a canonical world representation via information fusion of the 6D pose of the camera estimated via ORB SLAM 15 and RGB D images Kalman Filter KF is used to predict the 3D joints that lose track as well as smooth the joint trajectories Camera re localization in the SLAM is sensitive to the high dynamic environment induced by the moving human target and a lack of salient world features 16 Inspired by 17 we introduce a mask based strategy to enhance the robustness of the re localization accuracy Fig 1 demonstrates an overview of the proposed system for canonical pose estimation and gait analysis For validating the robustness of the system we compare the lower limb pose and joint angle estimation results with respect to the ground truth data from sixteen healthy volunteers The ground truth data were recorded with the Vicon motion capture IEEE Robotics and Automation Letters RAL paper presented at the 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 Copyright 2019 IEEE Fig 1 Overview of the proposed mobile 3D canonical gait analysis system system while subjects performed six gait patterns normal in toeing out toeing drop foot supination and pronation Next we investigate the potential of the proposed system to the abnormal gait recognition tasks In the recognition stage the Support Vector Machine SVM classifi er and the Bidirectional Long Short Term Memory BiLSTM 18 network were used to classify gait patterns This paper is organized as follows In Section II two steps of information fusion are described namely information fusion for 3D pose estimation and subsequently information fusion for human pose representation in the canonical coor dinate system Section III explains the gait feature selection and abnormal gait recognition Section IV describes several experiments to evaluate the accuracy and robustness of the proposed framework II RGB D BASED 3D CANONICAL GAIT ANALYSIS To construct a home based assistive system for gait anal ysis we used a light weight telepresence robot equipped with a single RGB D camera and without any additional sensing feedback To facilitate the applications on light weight mobile platforms with limited computing resources the captured RGBD images were live streamed to a remote workstation based on Real Time Streaming Protocol RTSP The robot exploited the 3D lower limb pose estimation to follow the human and collect data with simple control commands forward backward turn left turn right or the combination This control policy is also benefi cial for ex tending the proposed system across platforms A Information Fusion for 3D Lower Limb Pose Estimation Real time view invariant human pose estimation from RGB D images has attracted much attention in recent years 7 11 19 in which they used the Kinect sensor to estimate 3D skeleton both for gait analysis and action recognition The Kinect sensor was one of the fi rst systems that allowed real time 3D pose estimation and it is popular to the vision research community However its application in kinematic gait analysis provides unsatisfactory results and it is not suitable for clinical use 20 Moreover it only supports pose extraction from the live streamed data Zimmermann et al also demonstrated that human pose estimation by leveraging both rgb and depth images performs better than using depth data alone 21 To fuse information from depth images along with the abundance of texture features in RGB images we fi rstly use a Part Affi nity Fields approach that utilizes deep convolutional neural networks to detect human body parts and link them to a 2D skeleton 8 The 2D skeleton consists of several key joints and each joint j is represented as pj xj yj T where xj yj is the 2D pixel coordinates and indicates the prediction probability of this joint To acquire reliable depth value zjof joint j the holes in the raw depth images are fi rst fi lled Subsequently the bilateral fi lter in the spatial domain and the moving average fi lter in the temporal domain are adopted respectively To extract the 3D pose of the lower limb in real time we directly back project the 2D points onto the 3D space as Pj Xj Yj Zj based on the pin hole camera model Accordingly the raw 3D lower limb skeleton C S P1 Pj expressed in the camera frame C can be determined Furthermore the Kalman fi lter is adopted to predict the possible 3D joint position while its 2D position loses tracking which can also smooth the joint trajectories in 3D space Specifi cally the Kalman fi lter is applied for each joint respectively in which the state vector xKFconsists of the 3D position Pjand its velocity vector vj B Information Fusion for Human Pose Representation in the Canonical Coordinate System The 3D joints over time C S t are represented in the moving camera frame C To extract joint trajectories as well as gait parameters such as joints velocities gait speed and step stride length the 3D joint trajectory S t should be mapped to a canonical coordinate system E This requires to simultaneously extract information of the 6D trajectory XC t of the moving camera in relation to the surrounding environment Although the camera pose can be estimated by using additional sensors such as IMU and odometer this paper is focused on a single RGB D camera setup scenario Vision based SLAM algorithm is fi rst used to localize both the 6D camera pose XC t and the lower limb pose S t sequences in a canonical coordinate system E where E is determined by the initial pose of the camera in the Fig 2 Information fusion for human lower limb pose representation and 3D gait analysis in the canonical coordinate system The 3D map M of the environment is pre built and stored offl ine During the mobile gait analysis the 3D lower limb skeleton S is fi rstly estimated in the camera frame coordinate system C We utilize ORB SLAM to track the real time localization of the camera as well as the transformation matrix E T C indicating the transformation from camera space to a canonical coordinate system Accordingly the 3D human movement and the 6D camera motion can be represented in a canonical coordinate system E mapping stage of SLAM In this paper the 3D map M of the environment is pre built and stored offl ine by using the ORB SLAM 15 with RGB D input It allows the robot to relocate itself in an offl ine world representation of its environment However the recovery of camera pose in dynamic scenes is challenging 16 In the scenario of the robot following a human the human body typically occupies most of the image frame and it moves in a relatively fast pace which would re sult in unsatisfactory re localization in the pre built 3D map due to the lack of salient features To alleviate this problem and improve the camera re localization performance a mask based strategy is used A square human mask is generated in each frame according to the 2D joint estimation The mask not only covers the whole human body but also involves the neighbor area of the human body This is because the neighbor area is also infl uenced by the human movement and it is affected by motion blur and illumination vibration In the re localization module of the ORB SLAM algorithm features are only extracted from the image outside the human mask instead of the extraction from the whole image 15 This simple method can improve the stability in matching features of the current dynamic view with the pre built maps Given E XC and the pre built 3D map M E T C indicating the transformation from camera space to canonical coordinate system can be determined Finally the canonical human pose representation is derived by E S E T C C S 1 In this paper the transformation matrix C T R between camera space C and robot space R is assumed as the identical matrix I This canonical representation allows gait analysis to be performed in a view independent canonical space as demonstrated in Fig 2 thus providing the feasible Fig 3 Visualization of fi ve abnormal gait patterns a In toeing and out toeing gait patterns b drop foot gait c For the supination and pronation participants were wearing correction insoles which helps for simulating these two gait abnormalities in a natural manner d To capture the ground truth data by the Vicon motion capture system refl ective markers were attached to the human body and the camera estimation of the 3D joint trajectories and the corresponding gait indices heel and toe strikes gait speed and step length Moreover the spatial relationship among the human target moving camera robot and the 3D structure of the surrounding environment can be captured III 3D GAIT ANALYSIS FOR ABNORMAL GAIT RECOGNITION In this section we investigate the potential of the proposed system for abnormal gait recognition To this end we fi rst collected a gait database from sixteen subjects using a Vicon motion capture system and our proposed system Next the joint angle features were extracted from 3D lower limb skeleton Finally two recognition approaches were given by using the joint angles as input for abnormal gait recognition A Ground Truth Data Acquisition Sixteen healthy volunteers mean age 27 5 fourteen males and two females were recruited for this study Each volunteer was asked to walk with normal style and imitate the other fi ve abnormal gait patterns in toeing out toeing drop foot supination and pronation as shown in Fig 3 This allows us to examine the accuracy of our proposed method under conditions that resemble abnormal gait pat terns in a range of gait abnormalities It has been proven that foot pronation supination angles and inward outward rotation angles are the critical indices for reducing mechanical stress and avoid sports injuries and osteoarthritis 22 In toeing and out toeing gait indicate the foot forward direction point inward and outward instead of straight ahead during walking Pronation refers to the inward rotation of the ankle joint and supination indicates the outward roll Especially the correc tion insoles were provided to the participants to naturally simulate the pronation and supination gait patterns without exaggeration For the acquisition of ground truth gait data 22 refl ective markers were attached to the lower limb as demonstrated in Figs 3 d The Vicon motion capture system tracked the 3D positions of these markers with high precision and frequency 120Hz The ground truth values of the camera pose were also recorded by the Vicon system The timestamps of the images and the Vicon system were recorded for synchro nization The participants initiated the gait from different directions and then walked along the diagonal line of a sensing area of size 2m 3m A subject repeated each gait pattern for eight times and the total number of samples for each subject is 48 B Gait Parameter Extraction Among different type of gait parameters joint angles of the 3D lower limb correlate well to the joint kinematics in the gait periodic movement 2 6 They are also sensitive to gait abnormalities and constitute typical measures in clinical evaluation Recalling that slight differences exist between the marker positions of the ground truth data and the joint positions thus joint angles are more appropriate to evaluate the accuracy of the proposed method compared to the absolute difference between refl ective markers and detected joints We fi rst calculate joint angular features in relation to a human based local coordinate system H Based on this human coordinate system Sagittal Coronal and Transverse planes are determined as illustrated in Figs 4 a Let de note the 3D positions of the RASI and LASI joints at time t as PRASI t and PLASI t respectively The origin of the coordinate system H at time t is denoted as o t 1 2 PRASI t PLASI t Two unit vectors at time index t can be determined by V1 PRASI t PLASI t PRASI t PLASI t and V2 o t o t 1 o t o t 1 Accordingly H xh yh zh can be calculated by xh V1 yh V1 V2 V1 V2 zh xh yh xh yh 2 Commonly the 3D lower limb skeleton can be represented by six link segments Thigh L Shank L Foot L Thigh R Shank R and Foot R where L and R indicate left and right As demonstrated in Fig 4 a we fi rst calculate the joint angles x y z between each link segment l with respect to the normal vectors of the Sagittal Coronal Transverse planes respectively The angle between link l and fl oor normal vector nfis referred to f Note that the fl oor plane can be determined in the mapping process of the ORB SLAM Next the joint angles between two segments are also considered in this paper As illustrated in Fig 4 b the knee angle kneeis cal culated by the thigh and shank segments and the ankle angle ankleindicates the angle between shank and foot segments Finally another critical gait parameter which indicates the angle between the foot direction and walking path is the foot Fig 4 Illustration of the human body coordinate system and various extracted joint angle features for abnormal gait recognition a A human body coordinate system H can be defi ned as Eq 2 From this local coordinate system we can defi ne three planes as Sagitta