深度学习在图像处理中应用总结和优化.doc

深度学习在图像处理中应用总结和优化/kwotsin/awesome-deep-visionfrom /m2dsupsdlclass/lectures-labsLeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998).Gradient-based learning applied to document recognition.LeNetSimonyan, Karen, and Zisserman. “Very deep convolutionalnetworks for large-scale image recognition.” (2014) VGG-16Simplified version of Krizhevsky, Alex, Sutskever, and Hinton.“Imagenet classification with deep convolutional neural networks.”NIPS 2012 AlexNetHe, Kaiming, et al. “Deep residual learning for imagerecognition.” CVPR. 2016. ResNetSzegedy, et al. “Inception-v4, inception-resnet and the impactof residual connections on learning.” (2016)Canziani, Paszke, and Culurciello. “An Analysis of Deep NeuralNetwork Models for Practical Applications.” (May 2016).classification and localizationRedmon, Joseph, et al. “You only look once: Unified, real-timeobject detection.” CVPR (2016)Liu, Wei, et al. “SSD: Single shot multibox detector.” ECCV2016Girshick, Ross, et al. “Fast r-cnn.” ICCV 2015Ren, Shaoqing, et al. “Faster r-cnn: Towards real-time objectdetection with region proposal networks.” NIPS 2015Redmon, Joseph, et al. “YOLO9000, Faster, Better, Stronger.”2017segmentationLong, Jonathan, et al. “Fully convolutional networks forsemantic segmentation.” CVPR 2015Noh, Hyeonwoo, et al. “Learning deconvolution network forsemantic segmentation.” ICCV 2015Pinheiro, Pedro O., et al. “Learning to segment objectcandidates” / “Learning to refine object segments”, NIPS 2015 /ECCV 2016Li, Yi, et al. “Fully Convolutional Instance-aware SemanticSegmentation.” Winner of COCO challenge 2016.弱监督学习 Weak supervisionJoulin, Armand, et al. “Learning visual features from largeweakly supervised data.” ECCV, 2016Oquab, Maxime, “Is object localization for free? –Weakly-supervised learning with convolutional neural networks”,2015Self-supervised learningDoersch, Carl, Abhinav Gupta, and Alexei A. Efros.“Unsupervised visual representation learning by contextprediction.” ICCV 2015.dnn优化Ren, Mengye, et al. “Normalizing the Normalizers: Comparingand Extending Network Normalization Schemes.” 2017Salimans, Tim, and Diederik P. Kingma. “Weight normalization:A simple reparameterization to accelerate training of deep neuralnetworks.” NIPS 2016.Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton.“Layer normalization.” 2016.Ioffe, Sergey, and Christian Szegedy. “Batch normalization:Accelerating deep network training by reducing internal covariateshift.” ICML 2015GeneralizationUnderstanding deep learning requires rethinkinggeneralization, C. Zhang et al., 2016.On Large-Batch Training for Deep Learning: Generalization Gapand Sharp Minima, N. S. Keskar et al., 20161. A strong optimizer is not necessarily a stronglearner.2. DL optimization is non-convex but bad local minima andsaddle structures are rarely a problem (on common DL tasks).3. Neural Networks are over-parametrized but can stillgeneralize.4. Stochastic Gradient is a strong implicit regularizer.5. Variance in gradient can help with generalization but canhurt final convergence.6. We need more theory to guide the design of architecturesand optimizers that make learning faster with fewer labels.7. Overparametrize deep architectures8. Design architectures to limit conditioning issues:（1）Use skip / residual connections（2）Internal normalization layers（3）Use stochastic optimizers that are robust to badconditioning9. Use small minibatches (at least at the beginning ofoptimization)10. Use validation set to anneal learning rate and do earlystopping11. Is it very often possible to trade more compute