IROS2019国际学术会议论文集 1224

Toward model based benchmarking of robot components Gianluca Bardaro Mohamed El Shamouty Giulio Fontana Ramez Awad Matteo Matteucci Department of Electronics Information and Bioengineering Politecnico di Milano Italy Email name surname polimi it Fraunhofer Institute for Manufacturing Engineering and Automation IPA Germany Email name surname ipa fraunhofer de Abstract The results of scientifi c experiments performed by different groups are rarely directly comparable Efforts such as the European Robotics League offer to the community in the form of competitions well documented and stable benchmarks to assess the performance of existing systems However benchmarks can be equally useful at design time the Plug the one cited in the table i e the inverted pendulum is a benchmark widely used in robot control 16 TABLE I Benchmarks types with examples Open loopClosed loop Dataset basedany RAWSEEDS bench mark not possible Simulated coincides with Dataset based Inverted pendulum PhysicalERL s Object Perception FBM ERL s Navigation FBM V THE PLUG at each execution pass it receives a set point form the trajectory follower and relays it to the simulator which in turn provides updated positions and scans By acting as a man in the middle the benchmarking component can collect all the necessary information to compute the following performance metrics Time metric defi ned as the time elapsed in the simula tion from the start of the experiment to the arrival of the robot at the goal up to a timeout Distance metric defi ned as the mean distance between the actual trajectory of the robot and the target trajectory specifi ed by the benchmark The trajectory follower benchmark is closed loop because it requires the interaction of multiple components and the metrics are calculated by considering the evolution of the benchmark over time VII THE NEXT STEP BENCHMARK COMPOSITION As explained by Section II modelling benchmarks and introducing such models into the design process brings ben efi ts of its own which are available even to the designer of a single component However the benefi cial impact of model based benchmarking increases when the design concerns systems composed by multiple components In fact provided that i suitable benchmarks for each component of the system have been defi ned modelled and executed and ii the results of such execution are available then it is possible for the system designer to predict the performance of the system built with such components We use the term benchmark composition for such prediction process Nowadays the only tool available to the robot designer to predict system performance is simulation Simulation is powerful since it removes the need to physically build the robot and the experimental testbed on the other side it requires very accurate models of how each component works and how they interact among themselves and with their environment As the reliability of results increases so does the effort required to set up the simulation and the number of ad hoc operations needed to fi x specifi c issues so the most accurate simulations are also the less reusable Therefore reliable simulation is costly in terms of time and effort In autonomous robotics predicting the key performance elements for a system without having to simulate or even build it is mostly a dream Conversely in other fi elds of engineering this is the norm The tool that these other fi elds possess and robotics lack is called technical specifi cations This capability for prediction is typical of technical fi elds say electrical engineering that have a much longer history than robotics and where the way of building systems and especially of subdividing them into subsystem with well defi ned interfaces is standardised Once the boundaries the behaviour and the ways of interaction of the elements of a system are perfectly defi ned it is possible to describe each component with a set of technical specifi cations that tell the engineer what the key elements of its performance will be in a given context The technical specifi cations of a component are the results of standardised tests that investigate the aspects of its behaviour that are signifi cant for it to fulfi l its function within a system Technical specifi cations are in other words the outcome of a benchmarking process Model based benchmarking is a way for robotics to reach this desirable condition enjoyed by other disciplines Let us consider a system corresponding to a complete robot or a part of it Provided that the system is modelled as a set of interconnected components for each of which a model is available component models are exactly defi ned in their inputs outputs and behaviour for each component a set of performance elements that taken together fully characterise its performance as an element of the system have been defi ned for each performance element an objective benchmark 1685 a Trajectory planner benchmark b Trajectory follower benchmark Fig 2 Simplifi ed visualisation of the architecture of the two benchmarks as developed using the SmartMDSD toolchain has been defi ned for each benchmark a model has been defi ned that has matching interfaces to the inputs and outputs of the associated component model for each component the results of the actual execution of all benchmarks defi ned for it i e the technical specifi cations of the component are available then we posit that it is possible to use the system model to compose the results of the benchmarks concerning its individual components to reliably predict how the system will behave An example of such benchmark composition process is provided in Section VIII Technical specifi cations of components enable the engi neer at design time to make informed decisions about what components to use in their system and to understand the impact that such decisions have on system performance As said above this approach to system design is typical of other engineering fi elds but still lacking in robotics The main reason for this is that robot components hardware software mixed are not standardised and as a consequence their interactions interfaces data types etc are not stan dardised The same uncertainty affects the defi nition of what performance elements can act as signifi cant specifi cations Thus the performance elements that describe each non standard robot components are not well identifi ed and standard benchmarks to measure such performance elements are usually unavailable The underlying cause of this state of affairs is that au tonomous robots are complex robotics is young and we are still fi nding out the best ways to design and build them However this situation is changing as AI and robotics are moving towards mainstream technology As more and more advanced robots hit the market with self driving vehicles opening the way we expect robotics to gradually converge on what are the boundaries between the elements of a robot boundaries in turn defi ne components and their interfaces In this scenario it is now possible to work on standard models for robot components and turn robot design into a more structured and mature branch of engineering Conse quently it is now feasible to defi ne benchmarks right at the modelling stage then use such models to develop standard ised benchmarks that describe the real world performance of robot components in a meaningful way This is what Plug in the most extreme case the nature of the output of C1 may be incompatible with C2 By going back to the example let us assume the delivery robot has an Ackermann steering system This implies that the robot has a maximum steering angle therefore the trajectory provided by the planner must follow this requirement In a less extreme situation the performance of C1 infl uences those of C2 for instance when taking in account safety limitation A planner that minimise the length of the path has the side effect of creating a trajectory that is as close as possible to obstacles but in many application it is important for safety reasons to keep a certain distance e g social aware robot navigation In practice this can be done by defi ning performance met rics that can be used to parametrise the dependant component or that can be matched against its characteristics and require ments Moreover it is necessary to standardise confi guration parameters that are recurring for multiple components in our approach we use the concept of Benchmark Constraints see Subsection V A The aim of these constraints is two fold to frame the execution of the benchmark in a specifi c problem setting and to provide a confi guration profi le that can be used by composable components The importance of these kind of constraints is particularly evident in the navigation example the physical characteristics of the robot e g type of kinematic type and shape of the footprint etc needs to be the same for both the trajectory planner and the follower otherwise the provider component i e the planner cannot 1686 generate a suitable output for the dependant component i e the follower However the defi nition of the right metrics and constraints is not enough to grant the composability of two benchmarks and consequentially components The provider component will have different performances depending on its confi guration and only a subset of them are of interest for the dependant component It is necessary to create a library of benchmarks that explore a wide range of parameters combinations The result is doubly advantageous fi rst the provider developer can certify a reliable functioning range of his components second the dependant developer can choose the most suitable component and confi guration for his application Let us go back to the example introduced at the beginning of this section The provider component C1 a trajectory planner is defi ned using the following parameters cluttering robot footprint and kinematic type Cluttering ranges from 0 to 1 and defi nes how much contiguous free space is available in the environment the robot footprint defi nes the shape and size of the mobile base and the type of kinematic outlines the manoeuvrability of the robot i e differential drive Ack ermann or omnidirectional Given a specifi c confi guration the C1 component provides a trajectory characterised by a minimum maximum and average curvature The dependant component C2 a trajectory follower designed for Ackermann vehicles is defi ned by the following parameters maximum steering radius and robot footprint The performance mea sure of C2 is the maximum distance between the given trajectory and the followed one this measure is directly infl uenced by the nature of the input trajectory For example given a maximum steering radius there is a maximum curvature the autonomous vehicle can achieve Thanks to benchmark composition we can determine in advance if the two components are going to be compatible and if they are how the characteristics of C1 are going to infl uence the performance of C2 In particular we can estimate how well C2 will be able to follow the trajectory provided by C1 depending on the level of cluttering of the environment IX CONCLUSIONS In the last years benchmarking also through benchmark ing competitions such as the European Robotics League provided autonomous robotics with a way to objectively assess and compare the performance of systems and compo nents This ameliorates a historic limitation of this discipline namely a lack of key scientifi c properties such as compa rability repeatability and reproducibility in its experimental results Until today however benchmarking has not had any real impact on the design process for robots and components This paper shows how benchmarking can represent a useful element in the toolkit of the robot designer and how in corporating benchmark models in model based robot design brings about key benefi ts Project Plug Bench pursues this goal in the context of the RobMosys Ecosystem in particular by defi ning the Plug Bench Benchmark Meta model Model based benchmarking brings also another impor tant benefi t to robot design it provides a methodology benchmark composition to predict system performance from the performance of its components measured with suitable benchmarks Benchmark composition opens the way for autonomous robotics to become a more structured engi neering fi eld where component performance is effectively characterised by technical specifi cations which the designer can use to quickly select components and assess their effect on system performance ACKNOWLEDGMENT This work has been partially supported by the EU Com mission under the RobMoSys project via the Plug Bench project Authors are thankful to Enea Scioni for the fruitful discussions on the benchmarking meta models REFERENCES 1 P U Lima D Nardi G K Kraetzschmar R Bischoff and M Mat teucci Rockin and the european robotics league building on robocup best practices to promote robot competitions in europe in Robot World Cup Springer 2016 pp 181 192 2 2019 RoboticsCoordinationActionforEuropeTwo Online Available https www eu projects rockeu2 html 3 2019 The smart city robotic challenge sciroc Online Available https sciroc eu 4 2019 European robotics league Online Available https www eu league 5 G Bardaro A Semprebon and M Matteucci Aadl for robotics a general approach for system architecture modeling and code gen eration in IRC 2017 IEEE International Conference on Robotic Computing 2017 6 C Schlegel T Ha ler A Lotz and A Steck Robotic software systems From code driven to model driven designs in 2009 Inter national Conference on Advanced Robotics IEEE 2009 pp 1 8 7 S Dhouib S Kchir S Stinckwich T Ziadi and M Ziane Robotml a domain specifi c language to design simulate and d