IROS2019国际学术会议论文集 2329

Vision based Estimation of Driving Energy for Planetary Rovers using Deep Learning and Terramechanics Shoya Higa 1 Yumi Iwashita 1 Kyohei Otsu1 Masahiro Ono1 Olivier Lamarre2 Annie Didier1 Mark Hoffmann1 Abstract This paper presents a prediction algorithm of driving energy for future Mars rover missions The majority of future Mars rovers would be solar powered which would require energy optimal driving to maximize the range with limited energy The essential and arguably the most challenging technology for realizing energy optimal driving is the capability to predict the driving energy which is needed to construct an energy aware cost function for path planning In this paper we propose vision based algorithms to remotely predict the driving energy consumption using machine learning Specifi cally we develop and compare two machine learning models in this pa per namely VeeGer EnergyNet and Veeger TerramechanicsNet respectively The former is trained directly using recorded power while the latter estimates terrain parameters from the images using a simplifi ed terramechanics model and calculate the power based on the model The two approaches are fully automated self supervised learning algorithms To combine RGB and depth images effi ciently with high accuracy we propose a new network architecture called Two PNASNet 5 which is based on PNASNet 5 We collected a new dataset to verify the effectiveness of the proposed approaches Comparison of the two approaches showed that Veeger TerramechanicsNet had better performance than VeeGer EnergyNet I INTRODUCTION Despite the rapid maturation of autonomous driving tech nologies here on Earth there are many remaining challenges in this fi eld that are unique to extraterrestrial environments On Mars for example while the Curiosity rover and the upcoming Mars 2020 Rover are driven by a radioisotope thermoelectric generator RTG that can provide a constant level of power many potential future Mars rovers including the notional Sample Fetch Rover could be solar powered The operation of solar powered rovers is signifi cantly more complex because the available power is limited and uncon trollable The driving path is carefully chosen at the local scale 1 10 m to minimize the energy consumption which varies signifi cantly due to terrain type and topography The solar powered Mars Exploration Rovers MERs were often operated in a lily hopping mode where the rover s path and schedule are manually planned in a way to arrive at a spot with a favorable sunshine condition at the end of every These authors contributed equally to the paper 1Shoya Higa Yumi Iwashita Kyohei Otsu Masahiro Ono Annie Didier and Mark Hoffmann are with Jet Propulsion Laboratory California Institute of Technology 4800 Oak Grove Dr Pasadena CA 91109 USA shoya higa yumi iwashita kyohei otsu masahiro ono annie k didier mark k hoffmann jpl nasa gov 2Olivier LamarreiswithSTARSLaboratory Uni versityofTorontoInstituteforAerospaceStudies 4925DufferinSt Toronto ONM3H5T6 Canada olivier lamarre robotics utias utoronto ca Copyright 2019 All rights reserved 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 Fig 1 VeeGer predicts energy consumption at each candidate path to a goal Sol Martian day like a frog hopping between lily leaves on a pond The long range route is also chosen in consideration of power consumption while making sure to arrive on a Sun facing slope before the winter Automated energy aware driving of solar powered vehicles requires three capabilities estimation of energy consumption estimation of energy generation and path planning with time varying energy production This paper is concerned with the fi rst capability In particular to choose a path in an energy optimal way a rover has to remotely predict the energy required to drive on the terrain in front of it as illustrated in Fig 1 This forces us to make the prediction based on the vision The predicted driving energy will be incorporated into a cost map from which an optimized path is obtained The two major contributing factors to driving energy are terrain inclination and wheel terrain interaction For example driving on deep unconsolidated sand induces a large wheel sinkage wheel slippage causing greater friction between the wheels and the terrain Therefore the wheels on sand consume more energy than the ones on hard surfaces Such complex interplay between multiple factors make it extremely challenging to remotely predict the driving energy solely based on fi rst principles In this study we propose VeeGer Vision based Estimation of Expending and Generating Energy for Rovers algo rithm which predicts energy consumption at each candidate path Factors which change energy consumption are 2 fold i robot dependent factors such as velocity and weight ii environment dependent factors such as terramechanics types and slope Since the robot dependent factors can be controlled VeeGer focuses on estimating the environment dependent factors VeeGer is a fully automated self supervised learning al gorithm After training a machine learning model from collected datasets which include energy consumption and vision data VeeGer can predict energy consumption based IEEE Robotics and Automation Letters RAL paper presented at the 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 Copyright 2019 IEEE on vision specifi c data including RGB and depth images VeeGer consists of two approaches The fi rst one which we name VeeGer TerramechanicsNet is based on a ter ramechanics model approach The terrain parameters used for the model are preliminarily estimated by a convolu tional neural net CNN which predicts the wheel terrain interaction parameters from RGB and depth images The predicted parameters are further used to calculate energy consumption We propose a simplifi ed terramechanics model which reduces the number of parameters that need to be estimated compared to the original model The second one VeeGer EnergyNet estimates the energy consumption directly from RGB and depth images VeeGer EnergyNet skips the estimation of terramechanics parameters since a fully End to End learning VeeGer EnergyNet removes the human designed engineering model and thus can remove the human effort In VeeGer TerramechanicsNet and VeeGer EnergyNet to combine RGB and depth images effi ciently with high ac curacy we propose a new network architecture called Two PNASNet 5 which is based on PNASNet 5 1 In experi ments we compare the two VeeGer approaches with a test dataset newly collected by the Athena rover at the JPL Mars Yard The terrain parameters estimator which is used for training the VeeGer TerramechanicsNet is validated by a single wheel test To summarize the main contributions of this paper are Twofullyautomatedself supervisedlearningap proaches VeeGer TerramechanicsNetandVeeGer EnergyNet to predict energy consumption from images A simplifi ed terramechanics model The Two PNASNet 5 model which effi ciently com bines RGB and depth images Experimental verifi cation of terramechanics parameters and energy consumption with a newly collected dataset To the best of our knowledge VeeGer is the fi rst approach which predicts energy consumption for rovers from images only The rest of sections are organized as follows Section II reviews existing methods for terramechanics model based approaches and machine learning based approaches Sec tions III provides an overview of VeeGer Sections IV and V describe the details of the simplifi ed terramechanics model and VeeGer respectively Section VI presents a dataset collected with a single wheel testbed and the Athena rover Section VII explains experimental evaluations of the pro posed methods with the new dataset Finally Section VIII shows the conclusions and discusses future works II RELATED WORK A Terramechanics model based approaches The research fi eld that studies the wheel terrain inter action is called terramechanics 2 3 In the fi eld of terramechanics several works estimate terrain parameters or mobility performance under controlled environments In general parameter estimation and mobility performance es timation have been separately addressed so far and there is no work combining both as one application Iagnemma et al developed an online terrain parameter estimation algorithm using the telemetry data from MIT Single Wheel Testbed 4 Hutangkabodee et al 5 6 worked on soil parameter iden tifi cation based on Bekker s pressure sinkage relationship and Janosi and Hanamoto s shear stress equation 7 Ding et al also performed soil parameter identifi cation 8 Cross et al demonstrated that terrain parameters can be estimated using neural networks 9 Ojeda et al presented a current based slippage detection algorithm to estimate wheel slippage 10 Reina et al reported a wheel slippage and sinkage detection algorithm using an optical camera facing the side wall of a wheel 11 For the traveling performance estimation of the rovers Nagatni et al developed the drawbar pull estimation method using a built in force sensor array wheel 12 Although related works mentioned above were demon strated the estimation capabilities of the terrain parameter or the mobility performance separately there are no com bined approached that directly predicts the driving power consumption B Machine learning based approaches Machine learning techniques are used to improve tech nologies for rover autonomous navigation such as slippage prediction 13 and terrain classifi cation 14 15 Helmick et al proposed a system to classify surrounding terrain type and predict slip remotely with stereo cameras and IMU 13 Gonzalez et al proposed deep learning DL approaches for slippage prediction and terrain classifi cation 16 They com pared the results of the DL approaches with the ones of SVM and the DL approaches did not produce enough improvement compared with SVM One of the reasons might be that parameters of the DL network were trained from scratch In various applications including terrain classifi cation 17 18 and object classifi cation 19 1 utilizing transfer learning on models pre trained with huge datasets such as ImageNet 20 and then fi ne tuning hyperparameters on the new dataset brought huge improvements Wellhausen et al proposed a DL approach which predicts the ground reaction score encoding terrain properties 21 which is similar to our approach in VeeGer However they do not take into account slippage information which is crucial for predicting energy consumption They also utilize weakly supervised semantic segmentation in which a human manu ally gives annotations to generate training data Conversely our VeeGer uses self supervised learning where no human is involved for training data InbothVeeGer TerramechanicsNetandVeeGer EnergyNet we predict continuous values terramechanics parametersinVeeGer TerramechanicsNetandenergy in VeeGer EnergyNet from images Since most of DL networks trained with ImageNet are for classifi cation we extend a DL network for classifi cation to be utilized for regression Specifi cally we use PNASNet 5 1 which showed the best performance in ImageNet classifi cation VeeGer TerramechanicsNet Athena Rover Multi scale PNASNet 5 Large Driving Data 6 wheels 1 16 31 46 61 76 91 106 121 136 151 166 181 196 211 226 241 256 271 286 301 316 331 346 361 376 391 406 421 436 451 466 481 0 0 001 0 002 0 003 0 004 0 005 0 006 0 007 0 008 0 009 0 01 1 16 31 46 61 76 91 106 121 136 151 166 181 196 211 226 241 256 271 286 301 316 331 346 361 376 391 406 421 436 451 466 481 0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 Estimate Terramechanics parameters VeeGer EnergyNet Multi scale PNASNet 5 Large Driving Data 6 wheels Ortho projected Image RGB Ortho projected Image depth Fig 2 Overview of VeeGer which consists of VeeGer TerramechanicsNet and VeeGer EnergyNet Terramechanics Model Athena Rover Terramechanics parameters 1 16 31 46 61 76 91 106 121 136 151 166 181 196 211 226 241 256 271 286 301 316 331 346 361 376 391 406 421 436 451 466 481 0 0 001 0 002 0 003 0 004 0 005 0 006 0 007 0 008 0 009 0 01 1 16 31 46 61 76 91 106 121 136 151 166 181 196 211 226 241 256 271 286 301 316 331 346 361 376 391 406 421 436 451 466 481 0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 Driving Data 6 wheels Fig 3 Overview of the estimation of terramechanics parameters III OVERVIEW OFVEEGER For the proof of concept we use Athena rover Fig 1 a six wheeled robot with rocker bogie link suspension Each driving and steering motor has a small motor controller which provides motor telemetry including position velocity and motor current For the perception sensors it has a stereo mast camera for vision based navigation and terrain elevation assessment Although Athena has an inertia measurement unit IMU in this study the rover pose is predicted based on the stereo measurement Figure 2 shows an overview of VeeGer It consists of two phases i training phase for data collection and training CNN parameters of VeeGer TerramechanicsNet and VeeGer EnergyNet and ii on board phase for testing VeeGer TerramechanicsNet and VeeGer EnergyNet In the training phase after Athena rover captures RGB and depth images it moves along a path planned manually and records time series energy consumption along the path at each wheel Reina et al also created similar datasets for terrain assessment for precision agriculture 22 In VeeGer TerramechanicsNet terramechanics parameters at each wheel are estimated from the recorded power Fig 3 and these parameters are used to train the parameters of the CNN In VeeGer EnergyNet we directly use the recorded energy to train the parameters of the CNN In the on board phase after Athena rover captures RGB and depth images it calculates candidate paths to the next way point as shown with red lines in Fig 4 a where a candidate path was given by the operator in this study At each path we extract multiple patches which consist of ortho rectifi ed RGB and depth images Figs 4 b and c b c a Fig 4 a Examples of candidate paths red lines and example patch along the path b ortho rectifi ed RGB image of the patch and c ortho rectifi ed depth image of the patch The size of each patch is set as 1 m 1 m based on the size of Athena rover and the distance between patches is set 0 3 m We apply the trained CNN to predict energy consumption from these images IV SIMPLIFIEDTERRAMECHANICSMODEL For the sake of simplicity we assume a model for de formable terrain which is in general not applicable to hard surface such as rocks It is our future work to elaborate the model and adapt to hybrid terrains that contain both deformable and hard surfaces However our algorithm em pirically shows that the current simplifi ed model works with acceptable prediction accuracy A Force Torque Model of A Wheel On the deformable terrain the mobility performance of the wheel was modeled with the normal stress and shear stresses generated at the contact patch Fz r h b Z f r sin d Z f r cos d 1 Fx r h b Z f r cos d Z f r sin d 2 My r h 2b Z f r d 3 where Fz Fx and Myare the wheel load the drawbar pull and the wheel torque respectively As shown in Fig 5 r h represents the length including the wheel radius and the lug height b is wheel width f and rare the angle of the wheel rotation the entry angle to the terrain and the exit angle from the terrain respectively In Soil surface 0 in this study a0 0 4 and a1 0 3 were empirically set s is wheel slippage defi ned as follows s r h vx r h 4 where h is lug height is the angular velocity of the wheel and vxis ground velocity of the wheel B A Simplied Terramechanics Model and Parameter Esti mation To calculate the driving energy consumption of the rover effi ciently classical terramechanics models e g 2 24 7 are simplifi ed As described above the force and torque acting on the wheel are calculated with the normal and shear stress acting on the wheel terrain interface The normal stress shown in Fig 5 is simplifi ed as follows knormal z1 b n m f knormal z2 b n r m 5 where knormal is a coeffi cient z1and z2are the wheel sinkages separated into two parts at the maximum stress angle mas follows z1 r cos cos f 6 z2 r cos f r m r f m cos f 7 Since cohesion of the terrain is usually a small value Janosi s shear stress model 7 is simplifi ed as follows kshear pitch h 1 e j kx i 8 where is the shear stress kshear is a coeffi cient pitchis the pitch angle calculated based on the rover pose which is obtained from the terrain map is normal stress and kxis shear modulus in this study kx 0 005 was empirically set j is shear displacement and is calculated as follows 23 j r f 1 s sin f sin 9 Wheel sinkage can be classifi ed into static and dynamic sinkage and can be expressed as a function of slippage as follows z zs zds 10 where zsand zdare static sinkage and dynamic sinkage coeffi cients respectively Simplifi ed terrain parameters are obtained by solving the following optimization problem min knormal n kshear zs zd 1 F z W 2 1 My T 2 s t 0 0 knormal 9 0e5 0 1 n 1 0 0 0 kshear 1 0 1e 4 zs 2 r h 3 0 0 zd 0 0015 11 where W and T are an estimated wheel load and measured wheel torque respectively For training of the VeeGer TerramechanicsNet knormal n kshear zs and zdas variables and wheel slippage s in ground truth are used To solve the problem Sequential Least SQuares Pro gramming SLSQP algorithm was used under the condition of 1e 5of the stopping criterion and 1000 of maximum iterations As the ground true wheel load W calculated based on the rover pose estimation derived from the map and wheel torque calculated based on the wheel current and the motor model were used C Torque Power Conversion In this study we calculate the power of each driving motor on the basis of motor characteristics Athena rover wheel consists of a Maxon EC max 30 brush less motor and a harmonic drive whose gear reduction is 100 1 To convert to the motor power wheel motor current Iest is estimated as follow Iest Test iG kT m g 12 where iGis gear reduction kTis torque constant mand g are the ef