IROS2019国际学术会议论文集 0696

A Convolutional Network for Joint Deraining and Dehazing from A Single Image for Autonomous Driving in Rain Hao Sun Marcelo H Ang Jr and Daniela Rus Abstract In this paper we focus on a rain removal task from a single image of the urban street scene for autonomous driving in rain We develop a Convolutional Neural Network which takes a rainy image as input and directly recovers a clean image in the presence of rain streaks atmospheric veiling effect haze fog mist caused by distant rain streak accumulation We propose a synthetic dataset containing images of urban street scenes with different rain intensities orientations and haziness levels for training and evaluation We evaluate our method quantitatively and qualitatively on the synthetic data Experiments show that our model outperforms state of the art methods We also test our method qualitatively on the real world data Our model is fast and it takes 0 05s for an image of 1024 512 Our model can be seamlessly integrated with existing image based high level perception algorithms for autonomous driving in rain Experiment results show that our deraining method improves semantic segmentation and object detection largely for autonomous driving in rain I INTRODUCTION Driving in rain is challenging for both humans and au tonomous vehicles For autonomous vehicles their vision based perception functions e g object detection recognition and semantic instance segmentation require accurate feature learning of images of urban street scenes As the most common bad weather condition rain drastically degrades the visual quality of images and blocks the background objects These visibility degradations have negative impacts on image feature learning and cause many computer vision systems to likely fail In addition to autonomous driving many other applications such as outdoor surveillance systems also degrade when they are presented with images containing artifacts such as rain and haze These all make it a highly desirable technique to remove undesired visual effects caused by rain from images While rain streaks create the blurring effect which oc cludes and deforms the background scene distant rain streak accumulation generates the atmospheric veiling effect which further reduces the visibility In addition rain and fog often happen at the same time especially during heavy rain The effect of fog is signifi cant in images of street scenes and degrades the high level perception functions of autonomous vehicles In the last few decades many methods have been proposed for rain removal from a single image Despite the success of past methods most of them suffer from several limitations Hao Sun is with Singapore MIT Alliance for Research and Technology SMART sun hao smart mit edu Marcelo H Ang Jr is with Depart ment of Mechanical Engineering National University of Singapore NUS mpeangh nus edu sg Daniela Rus is with Massachusetts Institute of Tech nology MIT rus csail mit edu Fig 1 A derainig example of our deraining method which removes the rain streaks and haze for improving the visibility for autonomous driving in rain Left the input image with rain effects Right the output image of our method The effects of rain are complex Most of the deraining methods 1 5 only solve the effect of individual rain streaks without the consideration of the haze fog caused by rain streak accumulation Method 5 includes the global atmospheric light in their model but they do not solve it in their algorithm Methods 6 8 concatenate their proposed deraining models with existing dehazing methods which makes them not end to end Existing methods are slow and cannot be used for real time applications Furthermore existing methods are computationally expensive and many of them can only work for low resolution images which are not enough for high level perception Considering these limitations we propose an end to end Convolutional Neural Network CNN for deraining from a single image for autonomous driving in rain Our main contributions are 1 Our network takes a rainy image of the urban street scene as input and directly recovers a clean image Our model is able to remove the individual rain streaks atmospheric veiling effect haze fog mist caused by distant rain streak accumulation 2 Our network performs deraining and dehazing from the global context of an image Compared to past methods which separately estimate the parameters of the rainy model we optimize the model parameters jointly and generate a better solution 3 Our method is fast For an image of 1024 512 its processing time is only 0 05s It can be easily inte grated with existing image based high level perception algorithms for autonomous driving in rain 4 Based on CityScapes dataset 9 and Foggy CityScapes dataset 10 we propose a dataset containing synthetic rainy images of urban street scenes for training and evaluation Figure 1 shows an example of deraining results of our model 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 IEEE962 II RELATED WORK During the last few decades many methods have been proposed for rain removal They can be basically divided into two groups video based methods and single image based methods In particular single image based methods can be categorized into traditional methods and deep learning methods We briefl y review these methods as follows A Video Based Rain Removal For video based methods rain can be removed by lever aging the temporal information and analyzing the difference between adjacent frames Compared to single image based methods it is relatively easy to remove the rain from videos 11 14 In 12 they use the average pixel values from the neighboring frames to remove rain streaks Garg and Nayar et al 12 15 16 use photometric properties and temporal dynamics to describe rain streaks In 11 they detect and remove the rain streaks by minimizing the registration error between frames A review of video based deraining is presented in 17 B Traditional Single Image based Rain Removal Although video based methods work well they heavily rely on the temporal information of videos In this paper we focus on rain removal from a single image It is much more challenging but there is still much room for improvement In traditional methods kernel regression non local mean fi l tering dictionary learning Gaussian mixture model GMM and low rank representation are widely used In 18 kernel regression and non local mean fi ltering are used for rain streak detection and removal Dictionary learning is used in methods 3 19 22 Method 3 uses discriminative sparse coding for rain removal Method 20 decomposes a rainy image into a rain layer and a non rain layer and then sparse coding is applied to remove rain streaks from the rain layer In 23 GMM is used as a prior to separate the rain streaks C Deep Learning Single Image based Rain Removal Recently a lot of deep learning methods are proposed for single image based deraining and achieve superior perfor mances 2 4 8 24 The idea is to learn a mapping between input rainy images and their corresponding ground truths Method 24 focuses on raindrop and dirt removal Methods 2 8 apply guided image fi ltering 25 27 to decompose a rainy image into a detail layer and a base layer A CNN is applied for deraining in the detail layer Then the derained detail layer and base layer are combined to generate the output Their methods are far from real time performance Rain and fog often happen at the same time In 6 the authors propose a multi task deep learning model that learns the rain streak mask and the appearance of rain streaks They conduct deraining fi rst then conduct dehazing using 28 and conduct deraining again Method 7 proposes a convolutional and Recurrent Neural Network for single image deraining They perform deraining fi rst then use the dark channel method 29 for haze removal Many dehazing methods such as 28 30 31 also have the potential usage for deraining Generative Adversarial Network GAN meth ods such as 32 also have the potential usage for deraining however they usually generate unexpected artifacts in the output and not fast enough for real time processing III APPROACH Our network takes a rainy image as input and directly recovers a clean image In this section we fi rst present the physical model for the rainy image with the effects of nearby rain streaks and distant fog haze Then we present the CNN model for estimating the model parameters and generating the clean image We introduce a dataset containing syn thetic rainy images of urban street scenes for training and evaluation We integrate our model with existing semantic segmentation and object detection solutions for high level perception tasks of autonomous driving in rain A Rainy Image Model Our rainy image model is developed based on the classical atmospheric scattering model of the hazy image generation 33 34 Under rainy conditions rain streaks have various shapes and directions which occlude deform and blur the background scene Meanwhile distant rain streaks accumu late and generate haze fog effect which further reduces the visibility Our rainy image model takes consideration of all these effects and can be formally written as I x J x R x t x A 1 t x 1 where I x is the input rainy image and x indexes pixels in the image J x is the clean image to be recovered R x models the nearby rain streak effect A is the global atmospheric light which models the haziness level t x is the medium transmission which describes the light portion that is not scattered and reaches the camera t x is defi ned as t x e d x 2 where d x is the distance from the scene point to the camera and is the scattering coeffi cient of the atmosphere t x decreases as d x increases indicating that the longer distance from the scene point to the camera is the more rain streaks accumulate thus the larger haziness level and the less visibility are In order to recover J x from I x past methods estimate the values of t x R x A separately using techniques such as sparse dictionary and Gaussian mixture model They regard A as a global constant and set a value for A heuristically However global atmospheric light A and medium transmission t x are correlated and they should be learned together Estimating them independently provides a suboptimal solution the value of A is usually overestimated and overexposure can happen 28 30 To learn the global atmospheric light A jointly with the medium transmission t x inspired by 30 we model t x 963 R x A in two new values K1 x and K2 x Equation 1 is re expressed as J x 1 t x I x A 1 t x A R x 3 Equation 3 is then formulated as J x K1 x K2 x I x K1 x K2 x 4 where K1 x 1 t x I x A A I x 1 K2 x R x I x 1 5 In this way t x and A are learned jointly by estimating the value of K1 x while R x is learned in K2 x B Convolutional Joint Rain and Haze Removal To learn the values of K1 x and K2 x we design a CNN model which takes the rainy image as input outputs the optimal values of K1 x and K2 x from the global context of the image and generates the clean image through end to end learning Our network contains two branches where K1 x branch takes the input rainy image I x as input and outputs the optimal value of K1 x and K2 x branch takes I x as input and outputs the optimal value of K2 x After estimating the values of K1 x and K2 x the network generates the clean image using Equation 4 The K1 x branch is to learn the media transmission t x and the global atmosphere light A jointly by estimating the value of K1 x where both t x and A depend on the global scene We use dilated convolution to increase the receptive fi eld of our network in order to learn the contextual information While a larger receptive fi eld can encode more contextual features for learning it leads to coarse features for details To solve this we fuse features at different resolutions by concatenating network responses under different receptive fi elds The K2 x branch is to learn the value of R x by es timating the value of K2 x Similar to the K1 x branch multi receptive fi eld fusion is used for learning both global and local information We use batch normalization after each convolutional layer to alleviate the problem of proper initialization After each batch normalization layer the leaky rectifi ed linear unit is used as the activation function After generating the values of K1 x and K2 x we perform elementwise addition and multiplication to generate the clean image More details of the network are shown in Figure 2 During training we optimize the network parameters by minimizing the construction error mean squared error between the generated clean image and the clean ground truth We add additional supervisions on training the K2 x branch minimizing the construction error between the last convolutional map of K2 x branch and the ground truth rain mask The ground truth rain mask is generated when we synthesize the rainy data Conv 3 16 1 1 1 1 Conv 16 16 3 1 1 1 Conv 16 16 3 1 1 1 Conv 32 16 3 1 2 2 Conv 16 16 3 1 2 2 Conv 32 16 3 1 3 3 Conv 16 16 3 1 3 3 Conv 16 3 1 1 0 1 Conv 32 16 1 1 0 1 Conv 16 3 3 1 0 1 concate concate concate Conv 8 16 3 1 1 1 Conv 16 16 3 1 1 1 Conv 16 32 3 1 1 1 Conv 3 8 1 1 1 1 concate Conv 48 32 1 1 0 1 Conv 96 16 1 1 0 1 J x K1 x K2 x I x K1 x K2 x K1 BranchK2 Branch Rain Streak Mask Conv 8 16 3 1 2 2 Conv 16 16 3 1 2 2 Conv 16 32 3 1 2 2 Conv 8 16 3 1 3 3 Conv 16 16 3 1 3 3 Conv 16 32 3 1 3 3 K2 x R x I x 1 R x K1 x Fig 2 Network architecture Taking a rainy image as input our network directly recovers a clean image The left branch is the K1 x branch while the right branch is the K2 x branch The convolution parameters are shown as number of input and output fi lters kernel size stride padding dilation C Datasets It is very expensive to collect a large number of real world clean rainy image pairs for training and benchmarking our model Based on CityScapes dataset 9 and Foggy CityScapes dataset 10 we synthesize images with rain and haze for experiments The CityScapes images represent urban street scenes acquired by a vehicle camera in different cities Foggy CityScapes dataset is built based on the CityScapes dataset and simulates a collection of foggy images gener ated by their proposed fog simulation For each image of CityScapes dataset the Foggy CityScapes dataset provides three different versions of three different fog densities with a constant attenuation coeffi cient being 0 005 0 01 and 0 02 light medium and heavy haziness levels For each fog variant we use Photoshop 1 to create two different versions of varied rain streak intensities light and heavy rain For each rain intensity we further vary the rain streak orientation for 2 versions In total for each clean CityScapes image we generate 12 3 2 2 rainy images of different haziness levels rain streak intensities and directions The Foggy CityScapes dataset provides 8925 images for training 1500 images for validation and 4575 images for testing After augmenting the Foggy CityScapes dataset by introducing different variants with different rain features our synthetic dataset contains 35700 images for training 6000 images for validation and 18300 images for testing D Training We implement our approach in Pytorch We train our network on the synthetic data using Adam optimization with 1 964 MetricsFu et al 8 RESCAN 5 AOD Net 30 Ours PSNR12 8318 6315 3220 50 SSIM0 610 840 830 84 TABLE I QUANTITATIVE DERAINING EVALUATIONS ON SYNTHETIC RAINY IMAGES MetricsFu et al 8 RESCAN 5 AOD Net 30 Ours PSNR14 0320 0217 7021 56 SSIM0 640 880 870 88 TABLE II QUANTITATIVE DERAINING EVALUATIONS ON SYNTHETIC IMAGES OF LIGHT RAIN MetricsFu et al 8 RESCAN 5 AOD Net 30 Ours PSNR9 5616 5313 0219 16 SSIM0 520 790 790 80 TABLE III QUANTITATIVE DERAINING EVALUATIONS ON SYNTHETIC IMAGES OF HEAVY RAIN an initial learning rate of 10 3 weight decay of 0 0005 and momentum of 0 9 on a NVIDIA Titan X PASCAL GPU for 140 epochs We divide the initial learning rate by 10 at 60 and 80 epochs respectively The original image of our dataset has a dimension of 2048 1024 and due to computation limitation we resize the image to 1024 512 for both training and testing We only train our network on images with the heavy rain intensity with one rain streak orientation and all haziness levels light medium heavy We evaluate our network on images with all rain orientations intensities light heavy and haziness levels light medium heavy So there are 8925 images for training 6000 for validation and 18300 images for testing E Integration with High level Perception Algorithms Robust perception under rainy conditions is important for the safety and sustainability of autonomous driving Past deraining dehazing methods only focus on evaluating the image restoration performance and there is rare work to study the impact of rain and haze removal on high level perception tasks Method 30 integrates their proposed dehazing method with Faster RCNN 35 for object detection but their approach only focuses on hazy conditions In this paper we focus on a single image based rain removal problem of the urban street scene with the goal to apply this deraining model in real world autonomous driving Consequently in addition to evaluating our method on image recovering we also integrate our deraining model with an existing semantic segmentation algorithm and an existing object detection algorithm to analyze the effect of deraining on high level perception tasks in rain We derain the rainy image fi rstly and feed the processed image to an existing semantic segmentation algorithm PSPNet 36 or an existing object detection algorithm YOLOv3 37 Experiment results and discussions are provided in Section IV B IV EXPERIMENTS AND EVALUATION In Section IV A we benchmark our method with state of the art methods quantitatively and qualitatively on our synthetic rainy dataset We also evaluate our method qualita tively on the real world data In order to analyze the impac