IROS2019国际学术会议论文集 1736

Vehicular Multi Camera Sensor System for Automated Visual Inspection of Electric Power Distribution Equipment Jinsun Park1 Ukcheol Shin1 Gyumin Shim1 Kyungdon Joo1 Francois Rameau1 Junhyeok Kim2 Dong Geol Choi3and In So Kweon1 Abstract In this paper we present a multi camera sensor system along with its control algorithm for automated visual inspection from a moving vehicle To accomplish this task we propose a unique hardware confi guration consisting of a frontal stereo vision system six lateral cameras motorized to tilt and a GPS IMU sensor mounted on the roof of a car From the frontal stereo system we detect electric poles and estimate their corresponding 3D positions Based on this 3D estimation the tilt angles of the motorized lateral cameras are controlled in real time to capture high resolution images of the equipment typically installed a few meters above the road surface In addition inertial odometry information from the GPS IMU module is utilized for pose estimation object localization and re identifi cation among cameras Experimental results demonstrate the effi ciency and robustness of our system for automated electric equipment maintenance which can reduce human effort signifi cantly I INTRODUCTION The growing dependence on technology requires a stable electric power supply which has to be monitored inspected and maintained regularly to prevent operational failures In fact minor malfunctions in electric power systems may lead to environmental e g wild fi re traffi c congestion human e g large number of casualties or industrial e g shutdown of a plant disasters 1 2 Traditional inspections are typically conducted by humans with binoculars infrared cameras optical zoom cameras or helicopters However these methodologies are costly time consuming labor intensive and potentially dangerous for operators In order to overcome these problems vision based automatic inspection robots such as climbing robots 3 4 5 and Unmanned Aerial Vehicles UAVs 6 7 8 have been intensively investigated These vision based inspection robots improve effi ciency and accuracy reduce labor cost and minimize human exposure to potential risks While these robotic platforms have already demonstrated their effi ciency for transmission system inspection practical alternatives for distribution system inspection remain unexplored The power distribution system is the fi nal stage of power delivery transferring electricity from suppliers to individual 1J Park U Shin G Shim K Joo F RameauandI S KweonarewiththeRoboticsandComputerVisionLaboratory School of Electrical Engineering KAIST Daejeon 34141 Republic of Korea zzangjinsun shinwc159 shimgyumin jkd369 frameau iskweon77 kaist ac kr 2J Kim is with the Korea Electric Power Research Institute Ko rea Electric Power Corporation Daejeon 34056 Republic of Korea kim jh kepco co kr 3D G Choi is with the Department of Information and Communication Engineering Hanbat National University Daejeon 34158 Republic of Korea dgchoi hanbat ac kr consumers The power distribution in urban areas is par ticularly complex and often relies on hundreds of electric lines and poles closely intertwined In such an environment conventional inspection robots are inadequate and manual inspection by humans is still in use Therefore an automated visual inspection system suitable for urban environments is needed to reduce human effort while ensuring reliable inspection performance In this paper we propose a vehicular multi camera sensor system for automated visual inspection as shown in Fig 1 By adjusting the tilt angles of lateral cameras on the fl y the proposed system automatically captures multi view images of electric power distribution equipment such as electric poles insulators and transformers and detects those objects Throughout the paper we refer electric power distribution equipment as target object To perform these tasks we propose an object detection based geolocalization method and an algorithm to control our set of motorized cameras The proposed geolocalization algorithm detects electric poles using the frontal stereo vision system and then localizes detected objects in 3D GPS coordinates with the help of GPS IMU information Based on these geolocations and prior calibration of the inter camera poses the tilt angles of the lateral cameras are determined to capture the target objects The end products of this strategy are high resolution images of the target objects from various viewpoints for de tailed visual inspection We demonstrate the robustness and effi ciency of the proposed algorithm by synthetic simulations and real world experiments II RELATED WORK To ensure the safety and reliability of power transmission systems various inspection approaches have been proposed including helicopters climbing robots and UAVs 3 8 However each of these inspection systems has a limited range of applicability One of the most common inspection approaches remains the foot patrol inspection i e inspection by humans This type of inspection is typically conducted by a team of two inspectors traveling on foot to visually inspect electric power equipment and examine its status The main advantages of the foot patrol are its high accuracy based on expert human knowledge and its scalability as it can be applied to any kinds of power systems including power plants and transmission and distribution systems However high intensity labor for inspectors is inevitable and process is quite time consuming 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 IEEE281 a Example images of multiple viewpoint observation for target objects from the lateral cameras b Top view of the proposed sensor system c Detailed specifi cations of the proposed sensor system Fig 1 Illustration of the proposed multi camera sensor system and example images obtained by system One alternative to labor intensive foot patrol inspection is to utilize a climbing robot that can climb along the conductor and overcome obstacles in its path 3 4 5 9 10 Because the robot is directly adjacent to the power line and has very little vibration its main advantage is its high quality visual inspection output In addition this type of robot can carry more sensors and equipment than UAVs can and thus can provide various useful information On the other hand the main disadvantage is that it requires complex hardware confi gurations and electromagnetic shielding is essential Moreover its functionality must include reliable autonomous navigation along the conductor obstacle avoidance and automatic visual inspection For these reasons usage of this type of robot is limited to power line inspection of power transmission systems Another alternative method is UAV based inspection which inspects power lines and components while tracking the lines or pylons 6 7 8 11 12 UAVs provide high speed low cost and accurate inspection results with high mobility Thanks to its high mobility this approach can be applied to a wide range of power systems However there are many prerequisites for this approach such as precise positioning and attitude control camera stabilization light weight hardware and low computational costs If the positioning and attitude control fail the UAV may crash into the power lines and cause a serious accident Furthermore the quality of inspection is greatly affected by the limited hardware resources and external disturbances due to bad weather condition Although the aforementioned systems have reduced human labor to a certain degree and enabled effi cient automated inspections they are not suitable for the inspection of power distribution systems in residential and urban areas where numerous electric poles and lines are highly entangled Therefore a robust reliable and scalable automated visual inspection system suitable for this environment has to be de veloped For this purpose we propose a novel multi camera sensor system that can capture multiple viewpoint images for a target object by estimating optimal camera tilt angles based on object detection and object 3D geolocalization Compared with previous works our system is specifi cally designed to operate under this challenging scenario and we demonstrate its robust and reliable performance in synthetic and real world experiments III MULTI CAMERA SENSOR SYSTEM Most of the power distribution systems are typically installed alongside roads Therefore developing a visual inspection system from a vehicular platform is a relevant and practical strategy to facilitate infrastructure maintenance However such a strategy has yet attracted little attention because the majority of previous works have focused on robots or UAVs Therefore we propose a pioneering vehicle based multi camera sensor system designed to capture high resolution images of power distribution equipment such as electric poles transformers and insulators In a real world environment target objects are usually located at 20 m distance In order to obtain high resolution images of target objects a lens with long focal length is essential Because this kind of lens provides typically narrow fi eld of view our task is particularly complex in that the tilt angles of the lateral cameras have to be constantly adjusted in real time to align their fi eld of view with the target objects For this purpose our hardware confi guration consists of two frontal cameras 282 Fig 2 Pipeline of the proposed automated visual inspection system At each time step the vehicle pose is updated by GNSS aided inertial odometry After that electric poles detected by the frontal stereo system are geolocalized in GPS coordinates Finally optimal tilt angles of the lateral cameras are estimated based on geometric constraints and object detection results stereo six lateral cameras motorized to tilt and a GPS IMU sensor module 1Fig 1 shows the proposed sensor system that can be mounted on the roof of a car In this section we introduce each component of the system Frontal Stereo System Our system s stereo vision system is useful to estimate the 3D pose of electric poles with respect to the car Our stereo vision system consists of two cameras equipped with 5 mm focal length lenses These cameras are installed in front of a sensor base with a 40 cm baseline The stereo system is calibrated using a conventional approach 13 and synchronized by an external hardware trigger The left camera is set as a reference for the entire sensor system Therefore once the reference camera pose is expressed in GPS coordinates all the other cameras can be expressed in GPS coordinates Lateral Camera Groups The motorized lateral cameras are essential to capture high resolution images of power equipment These cameras are installed on both sides of the sensor base to inspect all the infrastructure surrounding the vehicle Three cameras with different viewing angles Fig 1 form a lateral camera group to cover a wide range of fi eld of view and obtain multiple viewpoints of the target equipment This design choice reduces multiple viewpoint image acquisition time signifi cantly to a few seconds especially compared to that of the foot patrol case a few minutes The cameras in each group are synchronized and each camera is mounted on top of a tilt motor for tilt angle adjustment For lateral cameras a 12 5 mm focal length lens is adopted because it provides suitable fi eld of view for visual inspection as shown in Fig 1 a Specifi cally our target objects are usually located at 20 m distance In such cases roughly 5 mm px resolution can be obtained The smallest target object e g insulator size is 300 mm therefore it appears at 60 px in an image which is a suffi cient size for object detection 14 Relative camera idle pose i e with no tilt to the reference camera is calibrated for each camera by hand eye calibration 15 Precisely the 1PointGrey BFS U3 51S5C C MicroStrain 3DM GX5 45 and DY NAMIXEL XL430 W250 T modules are adopted for our sensor system LEFTRIGHT Feature Extraction Region Fig 3 Object detection and geolocalization procedure Thanks to the stereo rectifi ed images from the stereo vision system feature extraction region can be effi ciently constructed from the detected bounding boxes After that the disparity of each object is estimated by stereo feature matching axis of tilt control is on the tilt motor not on the camera axis However we assume that this transformation is identity because the distance between those two axes compared to the distance to a target object is negligible Moreover due to the tilt angle refi nement step Sec IV C the error originating from this approximation can be easily compensated GPS IMU Module and Sensor Base The GPS IMU mod ule gives GNSS aided inertial odometry and NED North East Down orientation information which can be used for localization of the vehicle and the power equipment in GPS coordinates All sensor components including cameras motors and GPS IMU module are installed on a 650 500 mm sized rigid metal sensor base that can be mounted on the roof of a car We assume a rigid relationship among the sen sors including the lateral cameras at their home position i e no tilt The information from the GPS IMU module is utilized to estimate the pose of the reference camera in GPS coordinates because their relative transformation can be calibrated in advance 16 Note that the lateral cameras poses are also tracked thanks to the servo motor tilt control values IV OBJECTGEOLOCALIZATION ANDOPTIMALTILT ANGLEESTIMATION The proposed sensor system allows us to capture high defi nition images of power equipment However control of camera orientations remains complex because in a driving environment the distance between electric poles and the sensor system is not constant Moreover registering the captured images of equipment to a database DB containing locations and IDs of target objects which is managed by a power supply company is a highly desirable feature for regular maintenance and in depth inspection procedure To achieve the aforementioned tasks we propose an ob ject geolocalization technique and a lateral camera optimal tilt angle estimation algorithm These two algorithms are strongly correlated and improve their performance recipro cally First the object geolocalization algorithm provides the GPS referenced positions of the target objects Based on these results the proposed system identifi es matches and registers the target objects in the DB Moreover initial tilt angle of each lateral camera is estimated through object geolocation Then as the vehicle moves forward the tilt angle and object geolocation are refi ned using the optimal tilt angle estimation algorithm Fig 2 illustrates the entire procedure of our pipeline 283 Moving Direction plane Camera Frame World GPS Frame Image plane 1 1 plane a Initial tilt angle estimation b Detection based 3D refi nement c Motion based tilt angle refi nement Fig 4 Optimal tilt angle estimation a Initial tilt angle is determined from geometric constraints b 3D position of target object is refi ned using results of object detection c Initial tilt angle is refi ned based on predicted camera pose of next frame A Object Detection and Our Dataset One prerequisite of target object geolocalization is to detect an object in an image Although two stage object detectors 17 18 show performance superior to that of one stage detectors 19 14 they do not achieve real time performance To perform fast and reliable object detection we adopt YOLOv3 tiny 14 which requires low computa tional cost and shows competitive performance Since public object detection datasets 20 21 do not cover power distribution equipment we have constructed our own annotated image dataset containing both frontal and lateral viewpoints to train a universal detection network covering frontal and lateral viewpoints Our dataset consists of 1 457 images with 10 714 object bounding box annota tions including the 9 following classes ElectricPole Tree2 Transformer Switch InsulatorA InsulatorB InsulatorC In sulatorD COS Further details are available in Sec V B B Object Geolocalization For object geolocalization the stereo vision system is utilized to triangulate the detected electric poles This object localization procedure is briefl y described in Fig 3 First the pose of the vehicle is computed from the GPS IMU information and the transformation to the reference camera As a result the pose of the reference camera is expressed in GPS coordinates Simultaneously the electric poles are detected from the left image of the stereo system To perform 3D localization of the poles we extract keypoints and their ORB descriptors 22 in the detected object bounding boxes After that keypoints from each bounding box are matched to the keypoints from the right image To speed up the matching process we intentionally ignore the features above or below the bounding box since we assume rectifi ed images The disparity of a given bounding box is estimated from the matched features between stereo images i e mean disparity of the matches Finally each object is triangulated and geolocalized based on the current pose of the vehicle in GPS coordinates In addition to identify each electric pole 2Tree class is included to make an object detector robust to distinguish ElectricPole and Tree we search for and match the IDs of detected poles using a database that contains IDs and locations of electric poles of an administrative district We defi ne X as the 3D coordinates of the geolocalized electric pole in GPS coordinates To simplify the geolocalization and initial tilt angle estimation we rely on the common assumption that the elevation of power equipment on the pole is fi xed and known a priori Hpole 20 m C Optimal Tilt Angle Estimation As mentioned earlier the orientation of the lateral cameras should be continuously adjusted according to the vehicle motion and the 3D localization of the electric poles In order to tackle this prob