IROS2019国际学术会议论文集1827

Constructing a Highly Interactive Vehicle Motion Dataset Wei Zhan1, Liting Sun1, Di Wang2, Yinghan Jin3, and Masayoshi Tomizuka1 AbstractResearch in the areas related to driving behavior, e.g., behavior modeling and prediction, requires datasets with highly interactive vehicle motions. Existing public vehicle mo- tion datasets emphasize increasing the number of vehicles and time duration, but behavior-related researchers are suffering from two factors. First, strong interactions among vehicles are not well addressed and datasets are of relatively low- density to observe meaningful interactions. Second, most of the existing datasets are missing the map information with reference paths which is essential for driving-behavior-related research. To address this issue, a dataset with highly interactive vehicle motions is constructed in this paper. A variety of chal- lenging driving scenarios such as unsignalized intersections and roundabouts are included. Reference paths are also constructed from motion data along with high-defi nition maps so that key features can be generated for both prediction and planning algorithms. Moreover, we propose a set of metrics to extract the interactive motions in different maps, including the minimum difference of time to collision point (MDTTC) and duration of waiting period. Such metrics are used to quantify the interaction density of the dataset. We also give several representative results on prediction and motion generation utilizing the constructed dataset to demonstrate how the dataset can facilitate research in the area of driving behavior. I. INTRODUCTION Microscopic vehicle motion data is essential for driving- behavior-related research areas. Interactive motion data in challenging driving scenarios is highly demanded to enable autonomous vehicles to predict future motions of others 14 and to imitate expert or generate social behavior of human 57. Highly interactive vehicle motion data can also strongly facilitate the research on driver behavior modeling and analysis 810 as well as representation learning 11. There are typically two categories of sensor instrumenta- tion for collecting vehicle motion data, namely, birds-eye- view cameras and onboard sensors on data-collection vehi- cles. Datasets from birds-eye-view cameras, such as NGSIM 12 and highD 13 datasets, provide complete information for all vehicles in the recorded scene. Both of them have been intensively used and benefi ting both industrial and academic research. However, with only highway motion data included, they cannot provide more diverse motion data in different driving scenarios. 1W. Zhan, L. Sun, and M. Tomizuka are with Mechanical Sys- tems Control Laboratory (MSC Lab), Department of Mechanical En- gineering, University of California, Berkeley, CA 94720 USA (e-mail: ). 2D. Wang is with Xian Jiaotong University, Xian, P.R. China. This work was conducted during a visit to the MSC Lab at UC Berkeley. 3Y. Jin is with Zhejiang University, Hangzhou, P.R. China. This work was conducted during a visit to the MSC Lab at UC Berkeley. Fig. 1: An example of the highly interactive driving scenarios in the dataset. Many vehicles were entering the roundabout. The bounding boxes and object IDs of the vehicles are obtained via a detection network with data association and tracking algorithms. Motion datasets from onboard sensors can be grouped into two types. One type contains motions of a fl eet of data-collection vehicles with onboard GPS, such as 100- car study 14. Motions of surrounding entities impacting the behavior of data-collection vehicles are not included in such datasets. The other type of onboard-sensor-collected dataset includes motion data of surrounding entities from onboard LiDARs and front-view cameras, such HDD dataset 15. Various kinds of driving scenarios are included in the dataset. However, the information of surrounding entities for interaction is not complete due to limited fi eld of view of the sensors. Also, the repetitiveness of motion data for each driving scenarios is quite low, which makes it hard to capture the natural multi-modal driver behavior. Moreover, those datasets do not contain high-defi nition (HD) map information. For research on driving behavior, it is indispensable to have not only accurate physical layer in HD map, but also reference paths to generate accurate key features such as distance to merging point and lateral deviation. An effi cient and unifi ed reference path generation method is desired for various kinds of challenging scenarios. More importantly, highly interactive driving behavior is relatively sparse in the aforementioned datasets, although the total recording hours can be suffi ciently long. Therefore, to address all these issues, we propose to construct a dataset focusing on highly interactive vehicle motions in a variety of challenging driving scenarios with complete information 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Macau, China, November 4-8, 2019 978-1-7281-4003-2/19/$31.00 2019 IEEE6415 (a) SR(b) MA(c) GL (d) EP(e) FT Fig. 2: Highly interactive driving scenarios in the dataset. of surrounding entities and maps. Towards this goal, we try to explore answers to two fundamental questions: 1) how to automatically detect the interactivity among different entities? and 2) how to quantitatively defi ne the interaction intensity of a dataset? Answers to these questions are not trivial. In most cases, researchers manually identify and select the interacting pairs of entities from datasets. For instance, in order to obtain strong interaction pairs to create a benchmark for different prediction approaches, the authors of 16 manually selected highly interactive ramp merging motions from the NGSIM dataset. Automatic detection of such interactivity among entities can be hard, and there is, to the best of our knowl- edge, no solid existing method. In fact, there is even no ”ground truth” on whether vehicles are interacting or not at a specifi c moment. Metrics are expected to depict how strongly the vehicles interact. Regarding quantifying the interaction density of a given dataset, a simple summation of the total number of interactions might not be appropriate since interaction can not be strictly defi ned with supporting ground truth. Distributions of approximate metrics are expected to describe the interactivity of a dataset. Specifi c contributions of this paper are three-fold. First, accurate and complete vehicle motion data in various chal- lenging driving scenarios such as roundabouts and unsignal- ized intersections are provided from birds eye view. Second, methods to construct reference paths from data are provided along with the HD map to generated accurate key features for driving-behavior-related research. Third, metrics are pro- posed to describe the interaction intensity of vehicle motion pairs as well as interactivity of the dataset. Distributions of the proposed metrics for the dataset and exemplar motion pairs with strong interactions are provided. II. DATASETOVERVIEW Motion data in roundabouts and unsignalized intersections in Fig. 2, is collected using DJI Mavic 2 drones. The dataset contains vehicle trajectories processed from 15 hours of video recordings. Approximately 35000 vehicle trajectories are included. The raw videos are 4K (3840 x2160) by 30 Hz, and are downsampled to 10 Hz for further process. Methodologies for processing the data and constructing the HD map will be briefl y discussed in Section III. Videos are collected during busy hours at fi ve locations (SR, MA, GL, EP, FT shown in Fig. 2) with three roundabouts (SR, EP, FT), two all-way-stop interactions (MA and the one at the center of EP), and six branches of one-way-stop interactions (three in GL and three in EP). We will defi ne metrics to depict the interactivity of the motions to analyze the dataset and provide examples of highly interactive mo- tions pairs in Section IV. III. DATAPROCESSING ANDMAPCONSTRUCTION The methodologies of data processing, map construction and reference path generation from motion data are discussed in this section. A. Data Processing Typically, the videos collected by the drone are relatively stable within several seconds. However, when observing the videos within several minutes, the gradual drifting and rotation of the drone caused by winds and GPS drift can lead to obvious change of the static scenes. Moreover, 6416 there were gusts in which the drone cannot stabilize itself, and it can cause sudden drifting and rotation. Therefore, a video stabilization algorithm with transformation estimator is necessary. The video stabilizer projects every frame to the fi rst one with a similarity transformation. We employ a Faster R-CNN 17 trained by manually labeled images from our collected data. Satisfactory result can be obtained with rare inaccurate boxes, miss detection and false alarm. After obtaining the detection output, we employ a Kalman fi lter in the rough data association and tracking stage, which also mitigates the infl uence of the inaccuracies and smoothen the trajectories. An example of the output of the tracking algorithms are shown in Fig. 1. Examples of the fi nal paths and speed profi les are shown in Fig. 3 (a) and (b), respectively. The results are accurate comparing to the ground truth, and the paths and speed profi les are smooth. time (s) speed (m/s) (a)(b) Fig. 3: (a) Smoothened paths of the vehicles appeared in one minute in the 7-way roundabout. (b) Speed profi le of the vehicles appeared in one minute in the 7-way roundabout. B. Construction of Map and Reference Paths The high-defi nition (HD) maps are constructed via manual annotations from drone images and all the videos recorded in the same scenario are aligned with the corresponding map. The physical layer of the FT roundabout HD map is shown in Fig. 4 with curbs, lane markings and crosswalk boundaries. In order to make the learned models generalizable for similar scenarios with different geometry, Fren et Frame is often used instead of the Cartesian Frame. In Fren et Frame, the longitudinal and lateral motions of the vehicle can be partitioned, and the longitudinal motion may be generalized to scenarios with different geometry but similar topology 18. To facilitate the research utilizing Fren et Frame, all reference paths in each scenario are required. We will use FT which is the most complicated scenario in the sense of reference path, as an example to show how the reference paths are created from motion data. 1) Baseline Reference Paths: Fig. 4 (a) shows the baseline reference paths created for FT. The asterisks and rings represent the start and end points of the reference paths, respectively. We fi rst automatically extract all the trajectories in the dataset with the same entrance and exit branches and categorize them in the same cluster. Then for each cluster, the average path is obtained and then smoothened with a B ezier curve. (a)(b) Fig. 4: (a) Baseline reference paths, and (b) Segmented reference paths in the 7-way roundabout. 2) Segmented Reference Paths: The complete references shown in Section III-B.1 may not be usable for online motion prediction. When a vehicle is detected in the roundabout, in most cases it is diffi cult and unnecessary to determine its entrance branch. Predicting its future motions using the complete references, however, will assign multiple references with different entrance branches which might be sharing the same lanes but mildly different from each other in terms of waypoint coordinates. Such inconsistency will cause non- smooth predictions. Therefore, we propose to remove such redundancy by segmenting the paths to guarantee that there is only one segment of reference path within the same lane. Fig. 4 (b) shows the segmented reference paths created for FT. We fi rst create a reference circle shared by the merged baseline references. For all the baseline references which do not merge into the reference circle, the complete baseline references is still retained. For all the references merging into the reference circle, three segments are created, namely, the middle segment utilizing an arc from the reference circle, as well as the initial and ending segments from the baseline ref- erence with smooth merging and demerging to the reference circle. The interconnections between succeeding segments are guaranteed to be smooth via B ezier curve fi tting. IV. INTERACTIVITYDEFINITION ANDANALYSIS We propose a set of rules to extract the interactive be- haviors and describe the interaction intensity under different driving scenarios. The rules are established based on the semantic HD maps and different spatial representations of vehicle paths 18. As shown in Fig. 6, paths of the inter- active cars can be separated into two groups: (a) paths with static crossing or merging points, and (b) paths with dynamic crossing or merging points. A. Static Crossing or Merging Points As shown in Fig. 6 (a), interactive trajectories at inter- sections and roundabouts belong to this category. For each pair of trajectories with a static crossing or merging point, we fi rst use the semantic HD map to extract the default behavior for each vehicle, namely, the behavior of the vehicle when there are no other traffi c participants. Such default behavior of vehicles can be categorized into two groups: a) behavior without traffi c-rule enforced stops such as yield signs and b) behavior with traffi c-rule enforced stops such as stop signs. 6417 Fig. 5: Distribution of 4TTCminand the waiting periods across different driving scenarios in the dataset (a) static crossing/merging points (b) dynamic crossing/merging points Fig. 6: Topology of different interactive paths. In (a), the cross- ing/merging points between two paths are static and fi xed, while in (b), the crossing/merging points are dynamic. If the default behavior of both vehicles are free of traffi c-rule enforced stops, we calculate the time to collision point (TTC) at every time step (0.1s as the sampling time) for each car, and use the difference be- tween them (4TTC) as an indicator of the interactivity between the vehicles. The TTC at time t is calculated as follows: TTCt i = 4dt i 4vt i (1) with 4dt i is the distance of i-th cars location to the crossing/merging point along the road at time t, and 4vt i is the speed of i-th car at time t. The differ- ence in TTC is defi ned as 4TTCt=|TTCt 1 TTCt2|. When the minimum difference of TTC (MDTTC), i.e., 4TTCmin=min tTstart,Tend 4TTCtis smaller than a given threshold Tthresh,MDTTC , we can defi ne that interaction happens. Tstartand Tendare, respectively, the starting time index when both cars appear and the crossing time index when one of the cars passes the crossing point. If any of the vehicle is in default subjected to the traffi c- rule enforced stops, we use the behavior deviation from the default one as an indicator of the interactivity. For example, if a car stops at the stop line for more than a specifi ed threshold Tthresh,wait, and the other car is approaching or passing by in front of the stop line in the intersection/roundabout, we can defi ne that interaction happens. B. Dynamic Crossing or Merging Points As shown in Fig. 6 (b), interactive trajectories for lane change and ramp merging belong to this category. For such a scenario, we focus on interactivity between the lane- changing/merging vehicle (defi ned as the active vehicle) and following vehicle (defi ned as the responding vehicle) on the target lane. The indicator we use for the interactivity is again the minimum of 4TTCtover a period before the lane-crossing/merging happens. If 4TTCminis smaller than a given threshold Tthresh,MDTTC , we can defi ne that interaction exists. C. Distribution of Interactivity We also provide the distributions of 4TTCminand wait periods of all vehicles in the dataset over different driving scenarios. The results are shown in Fig. 5, where the x axis represents the period of 4TTCminand wait time in seconds, and y axis is the percentage over a specifi c scenario. We can see that the dataset contains highly interactive trajectories 6418 The interactive trajectories in the map 051015 Time (s) -60 -40 -20 0 20 40 longitudinal distance to collision points (m) The interactive trajectories in S-T domain car into the roundabout car in the roundabout The interactive trajectories in the map 051015 Time (s) -80 -60 -40 -20 0 20 40 longitudinal distance to collision points (m) The interactive trajectories in S-T domain car into the roundabout car in the roundabout The interactive trajectories in the map 051015 Time (s) -60 -40 -20 0 20 40 60 longitudinal distance to collision points (m) The interactive trajectories in S-T domain car into the roundabout car in the roundabout (a) (b) (c) Fig. 7: Examples of interactive trajectories with a high percentage of 4TTCmin 1s, and wait period greater than 3s. If both Tthresh,MDTTCand Tthresh,waitare set to 3 s, there are 10673 interactive pairs in the dataset. Some examples of the interactive behavior are shown in Fig. 7. In Fig. 7 (a), both cars initially approach the crossing point at similar speed, negotiating about who will go fi rst. At around t=5 s, the car into the roundabout (purple) decelerates and yields to the car already in the roundabout (green). In Fig. 7 (b), the green car accelerates to make the purple car yield to itself. On the contrary, the purple car in Fig. 7 (c) is more aggressive. It fi rst accelerates to negotiate with the green car, and fi nally makes the green car yield so that it can pass the crossing point fi rst. V. UTILIZATIONEXAMPLES ANDDATASETACCESS The dataset is intended to facilitate driving-behavior- related research areas mentioned in Section I. In this section, we will provide several examples to show how the proposed dataset