iros2019国际学术会议论文集pdf1366

1、2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Macau, China, November 4-8, 2019Cooperative Audio-Visual System for Localizing Small Aerial RobotsJose Rosa1 andMeysam Basiri213 or wireless positioning beacons 14. However, deploy- ing such systems, in the environment,

2、is not always efficient or even practical since these are confined in space and require deployment in advance of each mission. This work aims at obtaining a robust localization solution for micro-size aerial robots, operating alongside a ground robot while being independent of any external infrastru

3、ctures. This is achieved through cooperation with the ground robot that is carrying onboard vision and acoustics sensors, to obtain information about the position of the micro aerial robot. Such a system could also be used along with the aerial robot onboard vision based solutions to further enhance

4、 the robustness of position estimations.While vision based sensors are shown to be very success- ful and widely used in many robotic solutions for localization and tracking targets 4, 15, they suffer some limitations such as the dependency on illumination, visual contrast, the limited field of view,

5、 and the need for real-time processing of high resolution and high frame-rate images. For this purpose acoustic sensors are used in this work to complement the vision-based information and improve the robustness of position estimations. This is achieved by exploiting the already available sound wave

6、s from the engines of the micro aerial robots to extract additional pose information that does not suffer the same type of limitations as with the vision sensors 16, 17. This work follows the assumption that visual and acoustic sensors do not fail simultaneously.This paper describes an audio-visual

7、positioning system that can be used onboard of a ground robot operating in proximity of micro aerial robots. The system fuses visual and acoustic sensor data with other sensory information in real time, by employing a depth camera and a microphone array, allowing robust and reliable pose estimation

8、of the micro aerial robot. No similar results on position estimation using a depth camera and a microphone array were found in the literature. The method presented in this paper is not limited to the described objectives and could possibly be extended further to address other problems such as locali

9、zing and intercepting non cooperative noise-emitting aerial platforms.II. PROPOSED METHODThis section initially describes the vision-based and the audio-based measurement systems that rely on a Microsoft kinect v2 camera and an eight microphone array to measure the position of a micro aerial robot w

10、ith respect to a ground robot. Furthermore, a data fusion algorithm is presented to fuse the vision and audio measurements with the Inertial Measurement Unit (IMU) of the aerial robot (microquad- copter crazyflie 2.0 8). An extended Kalman filter (EKF) isAbstract Employing small size aerial robots,

11、acting as mobile airborne sensors, to work alongside ground robots can be extremely useful in many different robotic missions. Due to the strict constraints in terms of size, weight, 3D coverage, processing power and power consumption. There are not many technological possibilities for performing in

12、dependent self-localization for such tiny robots. This paper describes a cooperative audio-visual localization system to robustly estimate the position of a small aerial robot from a ground robot. Experimental results with a 40-gram quadrotor assess the performance of the system and demonstrate the

13、reliability gained through fusion of sound measurements with visual information.I. INTRODUCTIONEmploying small size aerial robots to work alongside ground robots can have important advantages in many differ- ent robotic missions 1, 2. Such robots can rapidly reach areas of interest by flying over ob

14、stacles and access areas that are inaccessible to the ground robots, and furthermore, provide elevated and birds eye view sensing of the ground robots and the environment 3, 4. This work falls under two bigger research objectives we are currently pursuing 1) to equip a domestic service robot with a

15、micro size aerial robot for facilitating personal domestic applications 5, 6 and2) to integrate unmanned ground vehicles and micro aerial robots for inspection of solar farm stations (H2020 Interreg Atlantic area EAPA 986/2018).Extremely small aerial vehicles with only few grams of weight have been

16、successfully developed over the few recent years allowing safe operation of these platforms in small indoor environments and in proximity of humans 7, 8. A key challenge imposed in using such tiny platforms as autonomous robots is to reliably measure their 3D positions in space. There exist very few

17、 technological possibilities that could provide onboard localization while satisfying the strict constraints of such small robots in terms of weight, size, power consumption, processing power, three dimensional coverage 9, 10, 11. In 10 we used a 4.7 gram camera to perform onboard vision based local

18、ization for a 40 gram quadrotor. However, the solution was highly dependent on illumination and available visual features in the environment. An alternative to onboard positioning is to use an external positioning system such as motion tracking cameras 12,This work was supported by the H2020 INTERRE

19、G ATLANTIC EAPA 986/20181Jose Rosa, Institute for Systems and Robotics of the Instituto SuperiorTecnico, Universidade de Lisboa, Portugal jose.rosaist.utl.pt2Meysam Basiri, Institute for Systems and Robotics of the InstitutoSuperiorTecnico, UniversidadedeLisboa,Portugalmeysam.basiritecnico.ulisboa.p

20、t978-1-7281-4003-2/19/$31.00 2019 IEEE6064used for data fusion and implemented in the Robot Operating System (ROS). EKF is a reliable approach for real-time state estimation 18, 19 that is widely utilized in sensor fusion as well as in position estimation of aerial robots 20, 21, 22.the quadcopter i

21、nside the camera field of view as well as preventing it to be too close to the camera (always at least a meter away). Figure 3 shows the error histograms for each axis measurement and a Gaussian fit for each histogram. The mean and standard deviation values can be analyzed in tableI. From figure 3 i

22、s seen that x and z axis present similar Gaussian fits where y has a higher mean absolute value as well as a higher standard deviation. This is due to the fact that y in the world frame corresponds to the camera depth (dp) which is not as precise as the camera image. It should also be noted that the

23、 depth sensor becomes less precise when the ball gets too close to the camera as it can be read in the literature 27. Finally, the visual solution achieved a 3D position estimation mean absolute error (MAE) of 0.013m.A. Visual Detection and EstimationAs previously stated, a kinect device was employe

24、d for vision-based estimation. This device is commonly present in ground robots as it is a low priced device equipped with a depth sensor which allows for three dimensional estimation. Kinect sensor has been integrated in various robotic projects, which facilitates its use by employing previously de

25、veloped tools such as the wrappers between the kinect sensors and the ROS environment 23 and 24. In future extensions of this work different and better performing cameras can be used, particularly for outdoor environments, where the Kinect camera lacks accuracy.As detection was not the main focus, a

26、 simple strategy was elaborated. Crazyflie detection was made using a red ping pong ball on top of the main frame. From 25, it was possible to see which colors have the least measurement deviations and red was the color chosen as it has little effect on kinect V2 measurements and is not a common col

27、or on the surrounding environment (to avoid false readings). Detection was based on color and shape, figure 1. Even though this method is not robust when there are more red objects present, it was sufficient for the proposed objective.Fig. 2. Kinect V2 3D measurementsFig. 1. Red ball detectionAfter

28、detecting the ball center in two dimension pixel coordinates (xp, yp), the pixel depth is determined dp. Using these values and the camera calibration matrix entries is possible to compute the real values(xr, yr), in meters, of the position in respect to the camera center as explained in the literat

29、ure 26.Fig. 3. Error histogram for xyz measurementsTABLE IGAUSSIAN FIT PARAMETERS FOR POSITION ESTIMATION ERROR IN METERSAfter computing the real coordinates(xr, yranddp)ofthe red ball center these coordinates were converted and published to the world frame.The following plots were made using the me

30、asurements obtained through the kinect along with the Optitrack motion capture system. In this way it was possible to evaluate measurements against the ground truth (Optitrack). ResultsB. Audio based Direction of Arrival EstimationIn this work, audio based localization is chosen as it is notably app

31、ealing by performing independently of visual positioning, i.e. acoustic data can be reliable whenevershow the visual position estimation by the depth camera to be reliable. Figure 2 was computed during flight while keeping6065meanstandard deviationx-0.001350.01019y-0.005120.01472z0.004080.01097visua

32、l information is not and vice-versa. As an example: the camera might be obstructed or the lighting may not allow camera detection which compromises visual measurements but not acoustic. Contrarily, if the environment is acoustically noisy it jeopardizes sound but not video. This subsection is dedica

33、ted to the design of an audio based direction of arrival (DOA) algorithm using a microphone array.Direction of Arrival is divided into two components: azimuth and elevation angles. Azimuth is measured in the xy plane and ranges from to while elevation is the angle between the xy plane and z axis and

34、 ranges from/2 to /2. Both these angles are measured in respect to the microphone array center frame.It was opted to build a microphone array composed by eight microphones. By having eight microphones is possible to design a geometry arrangement which, theoretically, al- lows unambiguity both in the

35、 vertical and in the horizontal plane. The array was also designed to be homogeneous, i.e., the accuracy is the same independently of where the sound source is positioned. Literature research suggests the microphone array should be shaped like a cube, in order to satisfy the intended specifications,

36、 where each microphone is positioned in the vertex, figure 4. From 28 is learned that, for an eight microphone array, the cube shaped distribution is the optimal solution in the case of far and near field conditions.at which measurements were obtained. The final decision was made based on the rate s

37、ince all other methods with the exception of SRP-PHAT were not acceptable for live estimation as they provided low output rates (below 5Hz). Moreover, in order to further improve resolution and sound detection rate the 3D space was restricted, based on the experiment constraints - as the aerial vehi

38、cle was only flied in front of the microphone array and kinect camera the range for azimuth only varies between -30 to 40 degrees and for the elevation between -30 to 40 degrees in respect to the microphone array position. All these methods were tested using 29 python package which allows for an eas

39、y comparison between all methods while supporting resolution and space limitations. Furthermore, in order to detect the sound made by the crazyflie, several experiments evaluating the frequency range of sounds produced were made, to posteriorly perform SRP-PHAT in the corresponding range. The follow

40、ing plots represent the direction of arrival estimations made under the previously described conditions. Similarly to the visual estimation, these measurements were compared with the ground truth obtained by the motion capture system Optitrack. Figures 5 and 6 show the obtainedvalues for azimuth and

41、 elevation angles, respectfully.InFig. 5. Azimuth measurementsFig. 4. Cube shaped Microphone Array - microphones highlightedThe microphones utilized are omnidirectional and are connected to a micro-controller which allows the feed to the computer to be made in eight different channels at a 48kHz sam

42、pling frequency.Having the literature and the methods available in consid- eration it was opted to employ the Beamforming technique Steered-Response Power Phase Transform (SRP-PHAT) as it provided the best results when compared with Multiple Signal Classification (MUSIC), Coherent Signal Subspace Me

43、thod (CSSM), Weighted Average of Signal Subspaces (WAVES) and Test of Orthogonality of Projected Subspaces (TOPS). These results were based on the average error when compared with the ground truth as well as with the rateFig. 6. Elevation measurementsthese figures is possible to see that the measure

44、ments follow the gist of the crazyflie movement yet there is a considerable amount of outliers as well as scattered measurements. Figure 7 represents an error histogram for the plots referenced above6066with its Gaussian parameters in table II where a considerable bias in elevation can be detected.

45、This is estimated to be derived from environment reflections. Nonetheless, the acoustic bearing estimation achieved mean absolute errors around 6deg.It is important to mention the position (xyz) is in respect to the global frame whereas the rest of the state vector is in the vehicles frame. As an ex

46、tended Kalman filter the system is described by the following equations from 30:(2)Xt = g(ut, Xt1)where t is the time instance, g in a non-linear function which describes the state transition model, u is the control input and E is the noise, assumed to be Gaussian. Since the control vector u is not

47、used in this project the non-linear functiondepends only on the previous statext1.For the systemmodel it was used a simple kinematics model which takesinto account the elements mentioned above. Nevertheless, it is necessary to apply a Rotation matrix, RG, tor transform the variables from the robot f

48、rame to the global frame.The measurement model is expressed by:Zt = h(Xt) + tZt = x, y, z, , , , , , , x, y, z, , (3)(4)Fig. 7. Error histogram for DOA measurementsTABLE IIGAUSSIAN FIT PARAMETERS FOR ANGULAR ESTIMATION ERROR IN DEGREESwhere Ztis the measurement vector at time instance t, h is a non-

49、linear function describing the measurement model and t is the measurement noise which is considered Gaussian as well. The majority of the observations are present in thestate vector(Xt),except for azimuth () and elevation ().The measurement function, h(Xt), matches the state model vector with the me

50、asurement vector. It is a 14 by 1 matrix where the first 12 entrances have a direct correspondence in the state vector - position, orientation, angular velocities and linear accelerations. The last two entries are the azimuth and elevation which depend on xyz.C. Extended Kalman FilterAs stated above

51、, the proposed objective is to develop a robust technique to aid small aerial vehicles to localize and navigate. From the subsection II-A it is possible to see that the depth camera provides very good results which can easily be used by themselves to achieve the suggested goal. However, aerial vehic

52、les might not always be in front of the camera or within its range which leads to the need of acquiringacousticdata. Subsection II-B impliesazimuthand elevation can be estimated using a microphone array even if presenting greater uncertainty. This chapter provides a tool toimprovesmallaerialvehiclep

53、ositionestimationwhenever the depth camera is not providing accurate measurements. An extended Kalman filter was designed since it is well studied and has proven to be efficient inthe field of Robotics. EKF allows the use of non-linear equations to describe both the system model and the measurements

54、 model. This is particularly important in the measurements as the DOA is described in the three dimensional space by a non- linear equation. Commonly, aerial vehicles are equipped with Inertial Measurement Units which provide information on the vehicles orientation, angular velocity and linear accel

55、eration. Since EKF is a tool to fuse multiple sensors data, it was opted to include IMU data as it could be of interest to further improve position prediction estimation. The extended Kalman filter system state vector X is composed by the position, the orientation, the linear and angular velocities

56、as well as the linear acceleration of the aerial robot.(5)(6)computeh(X )=t13,1p h(Xt)14,1 = arctan(z/ x +2 y ) 2The aim of the extended Kalman filter is toan ”approximation of true belief”, 30, by linearizing thenon-linear equations which describe the system. The belief bel(Xt) is represented by a

57、mean t and a covariance . In this work, the extended Kalman filter algorithm is composed by three steps:Prediction step, which computes the system expected mean and expected covariance based on the previ- ous estimate and system model behaviour.(7)(8)t = = G GT+ R ttt t1twhere G is the Jacobian of g, and R is the process noise covariance.Matching Step, where the measurements are compared to the predicted value in order to check if they are within a certain threshold. This step ensures EKF convergence as well as discards outliers. This step is applied sepa- rately to the visual and the a