IROS2019国际学术会议论文集0166

Pedestrian Density Prediction for Effi cient Mobile Robot Exploration Marc Patrick Zapf 1, Motoaki Kawanabe2, and Luis Yoichi Morales Saiki3 AbstractWe present a method to predict humans in unex- plored map areas given limited observations of the environment. We used a geometric representation of the environment based on cost maps and semantic room categorization. Human density distributions were generated using a human tracker based on LiDAR data recorded by a mobile robot. A Gaussian Process (GP) regression model was created to predict human density in surrounding unobserved map locations. GP prediction perfor- mance was evaluated on density data recorded in a series of ten simulations of 25 persons in an offi ce setting, and in real-world robot deployments in an offi ce-like environment. Experimental results demonstrate that the current method can predict human locations with an accuracy average of 70%. I. INTRODUCTION Mobile robots are frequently used to monitor human location in dynamic environments. Gathered data about humans can be used for robot navigation to avoid human frequented areas or monitor human activity. Restrictions in sensor range and resolution limit exhaustive human location monitoring especially in larger-scale environments. Installed ambient sensors such as laser range fi nders (LRFs) 13 could provide full coverage of the environment, however this permanent surveillance is expensive. Therefore, if we could effectively use the limited human observations by a mobile robot to build environmental models of human density, these models could allow inference on human density in so far unobserved locations. In this work we use Gaussian Process (GP) to interpolate and extrapolate human density distributions. Fig. 1 shows a scenario where a mobile robot with limited sensor range explores an offi ce environment with many rooms and desk compartments. First it detects and tracks humans (in red) and build human density maps of the room to the right on the top image. These maps, together with features describing room confi guration, are fed to a GP model which can predict human density distributions in unexplored areas. As the robot keeps exploring the environment entering the unexplored room on the left of bottom image, it updates the GP model *This work was supported by the Japan Science and Technology Agency through CREST project on artifi cial consciousness. Part of this work was supported by JSPS KAKENHI Grant Number JP16K21719. *This paper has supplementary downloadable material available at , provided by the authors. This includes an MP4 format movie clip, which shows application of the presented methodology on a mobile robot. 1Marc Patrick Zapf is with CR/RTC5-AP, Bosch (China) Investment Co. Ltd., P.R. Chinamarcpatrick.zapf 2 Motoaki Kawanabe is with the Department of Dynamic Brain Imag- ing, Advanced Telecommunications Research Institute International, Kyoto, Japankawanabeatr.jp 3Luis YoichiMoralesSaikiiswiththeNagoyaUni- versityInstituteofInnovationforFutureSociety,Japan morales yoichicoi.nagoya-u.ac.jp Detections Predictions Build GP regression model of human density Detect, track, and map humans Predict human locations Update density model EXPLORE UNEXPLORED EXPLORED Fig. 1: In the top, a robot explores the environment while it detects and tracks humans (in red). Human density maps are built and fed to a GP regression model. In the bottom the robot uses the model to predict human density in unexplored areas (in yellow) as it enters an unexplored room and keeps updating the model. and improves its prediction performance. Prediction of hu- man density (in yellow) in the middle room is done after having a model of rooms with similar confi guration. In this paper we present an approach to build human density maps from ongoing robot observations, and to utilize GP regression for iterative prediction of human density in so far unobserved locations. Unlike approaches predicting future human location directly from large sets of collected human trajectory time-series 46, our approach integrates all observed human trajectories into one continuously up- dated 2-D grid map, representing ”time-averaged” human density. In this work, we evaluate the performance of estimating human locations from the density maps. We use the maxima of the density maps as most likely locations of humans. We quantify GP prediction accuracy as (1) displacement of the predicted maxima from ground truth maxima in simulation, and as (2) frequency of encountering humans near predicted maxima in real-world experiments. To our knowledge, our work is the fi rst to interpolate time- averaged maps of human density using a GP model that includes features of room geometry and semantic category, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Macau, China, November 4-8, 2019 978-1-7281-4003-2/19/$31.00 2019 IEEE4615 and to use a kernel able to model human density distributions across multiple spatial scales. II. RELATED WORKS In this section previous works regarding human trajec- tory analysis and prediction using Inverse Reinforcement Learning (IRL), GPs, and neural networks are presented. Finally, we outline the limitations of previous works, and the distinction of our approach. Ziebart et al. 7 recorded human trajectories in a confi ned offi ce space via LRFs installed throughout the environment. Maximum-entropy IRL 8 was used on the collected data to forecast trajectories of newly detected humans. This approach relies on ubiquitous permanent surveillance of large parts of the environment and does not allow inference beyond the monitored areas. Henry et al. 9 applied IRL to learn human-like naviga- tion behavior from example trajectories created in a crowd simulator. The authors used GP Regression to model and map human density and traffi c fl ow direction on simulated crowded walkways. Human position and velocity were input as features. GP models were used to direct robot motion alongside human fl ow patterns. Ikeda et al. 10 modeled and mapped human fl ow along sub-goals in a shopping mall. The authors reasoned that humans, when heading to a fi nal goal point, pass a series of common landmarks, i.e. sub-goals, specifi c to the environment. However, sub-goals were only computed for monitored map portions, and the method relied on fi xed sensors in the environment. Limosani et al. 11 built human density maps from human observations of a mobile robot in an offi ce environment. Changes in room furniture and changes in human density were analyzed, however, model building was limited to a single room and no inference was made. Kucner et al. 12 modeled pedestrian fl ow maps from sparse pedestrian data using Gaussian Mixture Models for robot trajectory planning. Many works use Long-Short Term Memory Networks (LSTMs) 13 to forecast human trajectories several seconds into the future. Sun et al. 5 collected over 15,000 human trajectories in a care home environment using a robot- mounted LiDAR sensor over three months. LSTMs were trained with short-term pose observations and long-term temporal information. Pose of newly detected persons was predicted up to 9 seconds into the future. Pfeiffer et al. 4 forecast human trajectories from simulations and a real- world dataset up to 3 seconds into the future. To consider the effect of the persons surroundings on the chosen future trajectory, portions of the static map around the person as well as relative positions of other pedestrians were input to the LSTM. Recently, Vemula et al. 6 proposed an approach to forecast trajectories using LSTMs. Here, a graph-based representation to model spatial relationships between nearby pedestrians during prediction was used to reduce prediction error. Previous works such as Limosani et al. 11 have built maps of human density suitable for the scenario of Fig. 1. However, we also require prediction of human density Odometry Laser range finder Robot Map building (SLAM) Human detection & tracking Gaussian Process modeling Manual labeling Human density prediction Semantic map Geometric & cost map Human density map Fig. 2: System block diagram. Geometric obstacle maps are built and cost maps showing distance to obstacles as well as manually labeled semantic maps are created. Humans detected are mapped to human density maps. Human density as a function of grid cell coordinate, cost, and semantic category is used to train a GP model. Then human density in so far unobserved grid cells can be predicted. in relatively remote, previously unobserved locations and independent of the time the training trajectories were ob- served. IRL methods rely on optimal expert behavior in training data 14, 15, or require a fi nite state space and known dynamics 16. Neural network - based approaches primarily deliver short-term predictions for robot control and need relatively large prior observations before model training provides reliable estimates. GPs are a non-parametric method allowing for non-linear mapping of feature inputs to outputs, and permit inference on human density in unseen map areas. However, there are very few works estimating human density with GPs. In contrast to Henry et al.s 9 human fl ow - centered GP model, we include features of room geometry and semantic category to predict human density in unseen areas of large environments, from limited initial human trajectory data. Further, we use a rational quadratic kernel 17 which can model multiple spatial scales of human density distributions. We assess this approach both in customized simulations and real-world robot deployments. III. MAP BUILDING This section details creation of multi-modal maps which are used to train a GP model for human density prediction. Fig. 2 depicts the workfl ow. We used a mobile robot equipped with a 2D laser sensor for mapping, localization, and human tracking. For tracking we applied a human torso detector 11, 18. It consists of a laser clustering module detecting candidates of human torsos and a Kalman fi lter to track humans over time. The maximum reliable tracking distance was up to 6 meters. An RGB-D sensor was used for obstacle avoidance during navigation. GeometricgridmapswerebuiltusingtheROS slam gmapping package 19. These maps contain occupancy grid information of largely unchanging boundaries 20. Cost maps are based on geometric maps where the cost was modeled from distance to obstacles to a range 0.0,1.0. Semantic maps were manually labeled with six room cat- egories (free, desk space, offi ce, workspace, meeting room, leisure area). 4616 Geometric map Desk Free Office Leisure WorkMeeting Semantic map 0 Cost value N/A 254 Free space Unknown Cost map Fig. 3: Cost map and semantic maps of an offi ce environment with dimension 85 m x 82 m. Human density maps were built with the result of the tracks of the human tracker which were mapped into a grid map. A circular area of 0.5 m diameter is fi lled to refl ect human occupancy. To reduce false-positives, detec- tions 0% are displayed. The top fi gure shows an example of a map built by a robot. As the robot moves, only surrounding areas can be explored at a time. In the center, the corresponding ground truth density map is built directly from simulated pedestrian trajectories. In the bottom, human density predictions in unobserved locations are shown. ground truth map pairs were saved to disk every 30 seconds. Fig. 6 shows an example of a robot density map (top) and ground truth map (center) pair in simulation. Blue to red traces depict human density 0%. A GP model is built from the robot density map to predict human density in so far unobserved areas (Fig. 6, bottom). Light blue to green areas show predicted density 0%). We evaluated prediction accuracy by measuring displace- ment error between predicted and ground truth locations of maximum human density. We performed the following off- line analysis based on each robot and ground truth map pair saved to disk, i.e. in 30-second intervals: 1. GP model training and prediction: a GP model (section IV) was trained on the explored map section. Coordinates, cost, and semantics of seen grid cells were used as input and observed human density in those cells as training output. Human density was predicted in so far unobserved grid cells in the following areas (see Fig. 7): a) Interpolation within a convex hull around explored area t = 5 min Robot patrol area Prediction areas: Interpolation (”training”) Extrapolation 0-5 m Extrapolation 5-10 m t = 10 mint = 15 min Fig. 7: Map areas for GP training and prediction at t=5 min, 10 min, and 15 min relative to start of robot patrol. Data from one simulation run. Green: explored area. Red: prediction area in interpolation condition. Red+orange: prediction area for extrapolation 5 m distance to explored area. Red+orange+magenta: prediction area for extrapolation 10 m distance. b) Extrapolation up to 5 m distance to explored area c) Extrapolation up to 10 m distance to explored area These three conditions are to compare accuracy for density map interpolation vs. extrapolation. For condition a, a convex hull was constructed using the ConvexHull function of the SciPy spatial library 27, with explored grid cells as input, and were predictions made for all unexplored cells within the hull. For conditions b and c, all unexplored grid cells lying within the respective distance to at least one explored grid cell were selected for prediction. 2. Locate predicted and actual maxima of human density: Imagine a robot tasked to move to predicted human density maxima to spot humans. Then, even if the prediction is not 100% accurate, the predicted maximum should lie within sensor range to an actual maximum (i.e. ground truth from the simulator). Separately for prediction and ground truth maps, we determined local maxima in test areas 28. To avoid overly high spatial frequency of maxima, in case maxima were less than 5 meters apart only the highest-value one was selected. In case of equi-value maxima, DBSCAN clustering 29 between those maxima was performed and the centroid chosen as single maximum. Thus, for n yielded ground truth maxima, we only selected the n strongest predicted maxima to have equal numbers of maxima to compare. Maxima lower than 1% density were excluded to fi lter out very low densities. 3. Evaluate distance between predicted and ground truth maxima: We measured distance of each predicted maximum to the nearest ground truth maximum (Fig. 8). Distance along a shortest traversable path was computed to avoid pairing maxima lying in different rooms. Note that this is a stricter criterion yielding higher values than Euclidean distance. For each time step, median distance between predicted peaks and nearest ground truth maxima was calculated. B. Real-World Testing To evaluate real-world scenarios, we conducted visual inspection of prediction accuracy of human locations. The robot was deployed for eight runs for a maximum time of one hour each, looping between selected waypoints (Fig. 4). Four patrols took place in area A1, and four patrols in area A2 (slightly more frequented). The GP model was iteratively built and trained from the latest state of the explored grid map and corresponding 4619 Training data from robot patrol Prediction maxima Ground truth maxima (Median) distance between predicted and ground truth maxima Prediction in test areasGround truth in test areas 1.0 0.8 0.6 0.4 0.2 Human density Training area Test area Density 0 Local maxima detection Fig. 8: Process of comparing local human density maxima from test areas of prediction and ground truth maps. NAVIGATION AND MODEL BUILDING PREDICTION every 1-5 min 4 areas with predicted human density 1% Fig. 9: Real-world testing where the robot cycles waypoints. On the left side, the GP model is continuously updated based on observed human density. On the right side, human density was predicted in unobserved locations (squares). In this scene, prediction accuracy was 75% as in three of four areas humans were found. For ground truth, an experimenter inspected areas with predicted density 1%. human density map. Subsequently, human density in sur- rounding areas was predicted based on the latest model. GP modeling time increased with training data points, i.e. explored grid cell count. An experimenter went to inspect each map area where human density 1% was predicted. This threshold served to fi lter numerous areas of near-zero predicted human density. If at least one human was within the predicted area, the prediction was counted correct. If predicted areas covered multiple room compartments, we considered each room compartment as a separate prediction. The percentage of correct predictions was calculated by di- viding correct predictions by total number of predictions. For this evaluation, predictions were made for all locations within the rooms patrolled, thus at times requiring extrapolation by the GP. VI. RESULTS A. Simulations: Accuracy of GP Predictions Displacement errors between between predicted local maxima of human density and the nearest ground truth maximum are shown in Fig. 10 (interpolation condition). 0 0 5 10 15 20 25 Median distance (m) Displacement error between ground truth and predicted human density maxima (interpolation condition) Mean Single runs Patrol time elapsed (min) 50100150200250 Fig. 10: Median displacement error between predicted and ground truth local maxima of human density for interpolation condition. Colored lines show data for single runs as a function of patrol time. Average is depicted in black. 050100150200 250 Patrol time elapsed (min) 2 4 6 8 10 Median distance (m) Interpolation Extrapolation 5 m Extrapolation 10 m Mean displacement error between ground truth and predicted human density maxima (interpolation vs. extrapolation) Fig. 11: Compa