IROS2019国际学术会议论文集1224

Toward model-based benchmarking of robot components Gianluca Bardaro, Mohamed El-Shamouty, Giulio Fontana, Ramez Awad, Matteo Matteucci Department of Electronics, Information and Bioengineering, Politecnico di Milano, Italy Email: name.surnamepolimi.it Fraunhofer Institute for Manufacturing Engineering and Automation IPA, Germany Email: name.surnameipa.fraunhofer.de AbstractThe results of scientifi c experiments performed by different groups are rarely directly comparable. Efforts such as the European Robotics League offer to the community, in the form of competitions, well documented and stable benchmarks to assess the performance of existing systems. However, benchmarks can be equally useful at design time: the Plug the one cited in the table, i.e. the inverted pendulum, is a benchmark widely used in robot control 16. TABLE I: Benchmarks types, with examples. Open-loopClosed-loop Dataset-basedany RAWSEEDS bench- mark (not possible) Simulated(coincides with Dataset- based) Inverted pendulum PhysicalERLs Object Perception FBM ERLs Navigation FBM V. THE PLUG at each execution pass it receives a set-point form the trajectory follower and relays it to the simulator, which, in turn, provides updated positions and scans. By acting as a man-in-the-middle, the benchmarking component can collect all the necessary information to compute the following performance metrics: Time metric, defi ned as the time elapsed in the simula- tion from the start of the experiment to the arrival of the robot at the goal, up to a timeout. Distance metric, defi ned as the mean distance between the actual trajectory of the robot and the target trajectory specifi ed by the benchmark. The trajectory follower benchmark is closed-loop, because it requires the interaction of multiple components and the metrics are calculated by considering the evolution of the benchmark over time. VII. THE NEXT STEP: BENCHMARK COMPOSITION As explained by Section II, modelling benchmarks and introducing such models into the design process brings ben- efi ts of its own, which are available even to the designer of a single component, However, the benefi cial impact of model- based benchmarking increases when the design concerns systems composed by multiple components. In fact, provided that (i) suitable benchmarks for each component of the system have been defi ned, modelled and executed and (ii) the results of such execution are available, then it is possible for the system designer to predict the performance of the system built with such components. We use the term ”benchmark composition” for such prediction process. Nowadays, the only tool available to the robot designer to predict system performance is simulation. Simulation is powerful, since it removes the need to physically build the robot and the experimental testbed; on the other side, it requires very accurate models of how each component works, and how they interact among themselves and with their environment. As the reliability of results increases, so does the effort required to set up the simulation, and the number of ad hoc operations needed to fi x specifi c issues, so the most accurate simulations are also the less reusable. Therefore, reliable simulation is costly in terms of time and effort. In autonomous robotics, predicting the key performance elements for a system without having to simulate (or even build) it is mostly a dream. Conversely, in other fi elds of engineering this is the norm. The tool that these other fi elds possess and robotics lack is called: technical specifi cations. This capability for prediction is typical of technical fi elds (say, electrical engineering) that have a much longer history than robotics, and where the way of building systems -and especially of subdividing them into subsystem with well- defi ned interfaces- is standardised. Once the boundaries, the behaviour and the ways of interaction of the elements of a system are perfectly defi ned, it is possible to describe each component with a set of technical specifi cations that tell the engineer what the key elements of its performance will be in a given context. The technical specifi cations of a component are the results of standardised tests that investigate the aspects of its behaviour that are signifi cant for it to fulfi l its function within a system. Technical specifi cations are, in other words, the outcome of a benchmarking process. Model-based benchmarking is a way for robotics to reach this desirable condition enjoyed by other disciplines. Let us consider a system corresponding to a complete robot or a part of it. Provided that: the system is modelled as a set of interconnected components, for each of which a model is available; component models are exactly defi ned in their inputs, outputs and behaviour; for each component, a set of performance elements that (taken together) fully characterise its performance as an element of the system have been defi ned; for each performance element, an objective benchmark 1685 (a) Trajectory planner benchmark (b) Trajectory follower benchmark Fig. 2: Simplifi ed visualisation of the architecture of the two benchmarks, as developed using the SmartMDSD toolchain has been defi ned; for each benchmark, a model has been defi ned that has matching interfaces to the inputs and outputs of the associated component model; for each component, the results of the actual execution of all benchmarks defi ned for it (i.e., the technical specifi cations of the component) are available; then we posit that it is possible to use the system model to compose the results of the benchmarks concerning its individual components to reliably predict how the system will behave. An example of such benchmark composition process is provided in Section VIII. Technical specifi cations of components enable the engi- neer, at design time, to make informed decisions about what components to use in their system, and to understand the impact that such decisions have on system performance. As said above, this approach to system design is typical of other engineering fi elds, but (still) lacking in robotics. The main reason for this is that robot components (hardware, software, mixed) are not standardised, and as a consequence their interactions (interfaces, data types, etc.) are not stan- dardised. The same uncertainty affects the defi nition of what performance elements can act as signifi cant specifi cations. Thus, the performance elements that describe each (non- standard) robot components are not well identifi ed, and standard benchmarks to measure such performance elements are usually unavailable. The underlying cause of this state of affairs is that au- tonomous robots are complex, robotics is young and we are still fi nding out the best ways to design and build them. However, this situation is changing, as AI and robotics are moving towards ”mainstream technology”. As more and more advanced robots hit the market (with self-driving vehicles opening the way) we expect robotics to gradually converge on what are the boundaries between the elements of a robot: boundaries, in turn, defi ne components and their interfaces. In this scenario, it is now possible to work on standard models for robot components and turn robot design into a more structured and mature branch of engineering. Conse- quently, it is now feasible to defi ne benchmarks right at the modelling stage, then use such models to develop standard- ised benchmarks that describe the real-world performance of robot components in a meaningful way. This is what Plug in the most extreme case, the nature of the output of C1 may be incompatible with C2. By going back to the example, let us assume the delivery robot has an Ackermann steering system. This implies that the robot has a maximum steering angle, therefore, the trajectory provided by the planner must follow this requirement. In a less extreme situation, the performance of C1 infl uences those of C2, for instance, when taking in account safety limitation. A planner that minimise the length of the path has the side effect of creating a trajectory that is as close as possible to obstacles, but in many application it is important, for safety reasons, to keep a certain distance (e.g., social- aware robot navigation). In practice this can be done by defi ning performance met- rics that can be used to parametrise the dependant component or that can be matched against its characteristics and require- ments. Moreover, it is necessary to standardise confi guration parameters that are recurring for multiple components, in our approach we use the concept of Benchmark Constraints (see Subsection V-A). The aim of these constraints is two fold: to frame the execution of the benchmark in a specifi c problem setting and to provide a confi guration profi le that can be used by composable components. The importance of these kind of constraints is particularly evident in the navigation example, the physical characteristics of the robot (e.g., type of kinematic, type and shape of the footprint, etc.) needs to be the same for both the trajectory planner and the follower, otherwise the provider component (i.e., the planner) cannot 1686 generate a suitable output for the dependant component (i.e., the follower). However, the defi nition of the right metrics and constraints is not enough to grant the composability of two benchmarks, and, consequentially, components. The provider component will have different performances depending on its confi guration, and only a subset of them are of interest for the dependant component. It is necessary to create a library of benchmarks that explore a wide range of parameters combinations. The result is doubly advantageous: fi rst, the provider developer can certify a reliable functioning range of his components, second, the dependant developer can choose the most suitable component and confi guration for his application. Let us go back to the example introduced at the beginning of this section. The provider component C1, a trajectory planner, is defi ned using the following parameters: cluttering, robot footprint and kinematic type. Cluttering ranges from 0 to 1 and defi nes how much contiguous free space is available in the environment, the robot footprint defi nes the shape and size of the mobile base and the type of kinematic outlines the manoeuvrability of the robot (i.e., differential drive, Ack- ermann or omnidirectional). Given a specifi c confi guration, the C1 component provides a trajectory characterised by a minimum, maximum and average curvature. The dependant component C2, a trajectory follower designed for Ackermann vehicles, is defi ned by the following parameters: maximum steering radius and robot footprint. The performance mea- sure of C2 is the maximum distance between the given trajectory and the followed one, this measure is directly infl uenced by the nature of the input trajectory. For example, given a maximum steering radius, there is a maximum curvature the autonomous vehicle can achieve. Thanks to benchmark composition we can determine in advance if the two components are going to be compatible, and, if they are, how the characteristics of C1 are going to infl uence the performance of C2. In particular, we can estimate how well C2 will be able to follow the trajectory provided by C1 depending on the level of cluttering of the environment. IX. CONCLUSIONS In the last years, benchmarking (also through benchmark- ing competitions such as the European Robotics League) provided autonomous robotics with a way to objectively assess and compare the performance of systems and compo- nents. This ameliorates a historic limitation of this discipline: namely, a lack of key scientifi c properties such as compa- rability, repeatability and reproducibility in its experimental results. Until today, however, benchmarking has not had any real impact on the design process for robots and components. This paper shows how benchmarking can represent a useful element in the toolkit of the robot designer, and how in- corporating benchmark models in model-based robot design brings about key benefi ts. Project Plug&Bench pursues this goal in the context of the RobMosys Ecosystem, in particular by defi ning the Plug&Bench Benchmark Meta-model. Model-based benchmarking brings also another, impor- tant benefi t to robot design: it provides a methodology (benchmark composition) to predict system performance from the performance of its components, measured with suitable benchmarks. Benchmark composition opens the way for autonomous robotics to become a more structured engi- neering fi eld where component performance is effectively characterised by technical specifi cations, which the designer can use to quickly select components and assess their effect on system performance. ACKNOWLEDGMENT This work has been partially supported by the EU Com- mission, under the RobMoSys project, via the Plug&Bench project. Authors are thankful to Enea Scioni for the fruitful discussions on the benchmarking meta-models. REFERENCES 1 P. U. Lima, D. Nardi, G. K. Kraetzschmar, R. Bischoff, and M. Mat- teucci, “Rockin and the european robotics league: building on robocup best practices to promote robot competitions in europe,” in Robot World Cup.Springer, 2016, pp. 181192. 2 (2019)RoboticsCoordinationActionforEuropeTwo. Online.Available:https:/www.eu- projects/rockeu2.html 3 (2019) The smart city robotic challenge (sciroc). Online. Available: https:/sciroc.eu/ 4 (2019) European robotics league. Online. Available: https:/www. eu- league/ 5 G. Bardaro, A. Semprebon, and M. Matteucci, “Aadl for robotics: a general approach for system architecture modeling and code gen- eration,” in IRC 2017-IEEE International Conference on Robotic Computing, 2017. 6 C. Schlegel, T. Haler, A. Lotz, and A. Steck, “Robotic software systems: From code-driven to model-driven designs,” in 2009 Inter- national Conference on Advanced Robotics.IEEE, 2009, pp. 18. 7 S. Dhouib, S. Kchir, S. Stinckwich, T. Ziadi, and M. Ziane, “Robotml, a domain-specifi c language to design, simulate and deploy robotic applications,” in International Conference on Simulation, Modeling,