人工智能与机器学习课件

1、深度学习快速推进中的机器学习与人工智能前沿提纲深度学习(DL)及其应用前沿DL在CV领域应用的启示关键算法介绍Perceptron及学习算法MLP及其BP算法Auto-EncoderCNN及其主要变种关于DL的思考与讨论2机器学习的基本任务3Class label(Classification)Vector(Estimation)dog, cat, horse, Object recognitionSuper resolutionLow-resolution imageHigh-resolution image源起生物神经系统的启示神经元之间通过突触(synapse)连接层级感受野，学习使突触

2、连接增强或变弱甚至消失4Hubel, D. H. & Wiesel, T. N. (1962)第一代神经网络5Frank Rosenblatt(1957), The Perceptron-a perceiving and recognizing automaton. Report 85-460-1, Cornell Aeronautical Laboratory.第一代神经网络单层感知机(Perceptrons)模型的局限性Minsky & Papert的专著Perceptron(1969)只能对线性可分的模式进行分类解决不了异或问题几乎宣判了这类模型的死刑，导致了随后多年NN研究的低潮62n

3、d Generation Neural Networks多层感知机(Multi-layer Perceptron, MLP)超过1层的hidden layers（正确输出未知的层）BP算法 Rumelhart et al., 1986Compute error signal; Then, back-propagate error signal to get derivatives for learning7David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). Learning representa

4、tions by back-propagating errors. Nature323(6088): 533536Error BackpropagationW is the parameter of the network; J is the objective functionFeedforward operationBack error propagationDavid E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). Learning representations by back-propagat

5、ing errors. Nature323(6088): 533536Output layerHidden layersInput layerTarget values2nd Generation Neural Networks理论上多层好两层权重即可逼近任何连续函数映射遗憾的是，训练困难It requires labeled training dataAlmost all data is unlabeled.The learning time does not scale wellIt is very slow in networks with multiple hidden layers.

6、It can get stuck in poor local optimaThese are often quite good, but for deep nets they are far from optimal.91990-2006更流行Specific methods for specific tasksHand-crafted features (SIFT, LBP, HOG)ML methodsSVMKernel tricksBoostingAdaBoostkNNDecision tree10Kruger et al. TPAMI13A Breakthrough Back to 2

7、0062006年，通过分层的、无监督预训练，终于获得了训练深层网络结构的能力11A Breakthrough Back to 2006Hinton, G. E., Osindero, S. and Teh, Y., A fast learning algorithm for deep belief nets. Neural Computation 18:1527-1554, 2006Hinton, G. E. and Salakhutdinov, R. R. (2006) Reducing the dimensionality of data with neural networks. Sci

8、ence, Vol. 313. no. 5786, pp. 504 - 507, 28 July 2006Yoshua Bengio, Pascal Lamblin, Dan Popovici and Hugo Larochelle, Greedy Layer-Wise Training of Deep Networks, Advances in Neural Information Processing Systems 19 (NIPS 2006)MarcAurelio Ranzato, Christopher Poultney, Sumit Chopra and Yann LeCun. E

9、fficient Learning of Sparse Representations with an Energy-Based Model, Advances in Neural Information Processing Systems (NIPS 2006)12其实是有例外的CNN卷积神经网络CNNK. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biolo

10、gical Cybernetics, vol. 36, pp. 193202, 1980Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541551, 1989Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner, “

11、Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 22782324, 199813其实是有例外的CNNNeocognitron 198014K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological

12、 Cybernetics, vol. 36, pp. 193202, 1980Local Connection例外：CNN用于数字识别15例外：CNN用于目标检测与识别16而且，东风同样重要大数据大数据大数据语音图像视频计算能力并行计算平台GPU大量部署开放的社区开源，开放数据17语音识别(2011)1819862006DBNScienceSpeech2011BP2012年计算机视觉的巨大进步ImageNet物体分类任务上物体分类任务：1000类，1,431,167幅图像1919862006DBNScienceSpeech20112012RankNameError rates(TOP5)Des

13、cription1U. Toronto0.153Deep learning2U. Tokyo0.261Hand-crafted features and learning models.Bottleneck.3U. Oxford0.2704Xerox/INRIA0.271BPImageNet with Deep CNN方法：大规模CNN网络20A. Krizhevsky, L. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NIPS, 2012.Im

14、ageNet with Deep CNN方法：大规模CNN网络650K神经元, 60M参数Trained with BP on GPU使用了各种技巧+dropoutReLU, Data augment, contrast normalization,.被Google收编(Jan 2013)Google+ Photo Tagging(2013.5)21A. Krizhevsky, L. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NIPS, 2012

15、.ImageNet物体分类(2013)1000类，1,431,167幅图像，Top 5错误率2219862006DBNScienceSpeech20112012RankNameError rates(TOP5)Description1NYU0.11197Deep learning2NUS0.12535Deep learning3Oxford0.13555Deep learning2013BPMIT Tech Review坐不住了23ImageNet物体分类(2014)1000类，1,431,167幅图像，Top 5错误率2419862006DBNScienceSpeech20112012Ran

16、kNameError rates(TOP5)Description1Google0.06656Deep learning2Oxford0.07325Deep learning3MSRA0.08062Deep learning20132014BPImageNet物体分类(2014)GoogLeNet CVPR201522个卷积层Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. CVPR 20152519862006DBNScienceSpeech20112012BP20142013ImageNet物体分类(2010-2

17、014)ImageNet Top 5 Error Rate上的持续进步2619862006DBNScienceSpeech20112012BP20142013ImageNet物体检测任务200类，456, 567 幅图像，检测率27传统方法SIFT+BOW+SPM深度方法R-CNN+GoogLeNet19862006DBNScienceSpeech20112012BP20142013物体分割/语义标注进步迅速Jonathan Long, Evan Shelhamer, Trevor Darrell. Fully Convolutional Networks for Semantic Segme

18、ntation. CVPR201528DL有多热Deep Learning for Vision 602篇文章中，仅标题中出现Deep的就有87篇，出现Convolution的47篇，出现Neural的40篇，出现Network的51篇，Recurrent 7篇Going deeper，优化，无监督、自主学习Fully Convolutional Network（for segmentation等）Vision and Language(for看图说话, Google, Fei-fei, Microsoft, UCB)RNN with LSTM（for 时序处理）Deep Learning f

19、or X(detection, metric learning, attribute, hash,)29计算机视觉的重大进步Vision and Language(Google, Microsoft, UCB)看图说话：Minsky 60年前布置的作业30Show and Tell: A Neural Image Caption Generator (a work from Google)From Captions to Visual Concepts and Back (a work from Microsoft)Long-term Recurrent Convolutional Netwo

20、rks for Visual Recognition and Description(a work from UTA/UML/UCB)人脸识别上的进步正确率95.17% D.Chen, X. Cao, F. Wen, J. Sun, CVPR13正确率97.35% Y.Taigman, M. Yang, M.Ranzato, L. Wolf, CVPR14正确率99.47% Y. Sun, X. Wang, and X. Tang, CVPR14正确率99.63% F. Schroff, D. Kalenichenko, and J. Philbin, CVPR153119862006DBNS

21、cienceSpeech20112012Face20142015BP在LFW上，过去2年错误率从5%下降到0.5%(错300对错30对)人脸识别上的进步Labeled Face in the Wild (LFW)非限定条件下的人脸识别数据来源于因特网国外名人，Yahoo新闻广为人知的测试模式训练集：无限制验证任务测试集共6000图像对32Huang G B, Ramesh M, Berg T, et al. Labeled faces in the wild: A database for studying face recognition in unconstrained environme

22、nts. Technical Report, University of Massachusetts, Amherst, 2007.人脸识别上的进步2014: DeepFace 1 (Facebook)大数据：4K人，4.4M图像331 Taigman Y, Yang M, Ranzato M A, et al. Deepface: Closing the gap to human-level performance in face verification. CVPR, 2014.2 Sun Y, Wang X, Tang X. Deeply learned face representat

23、ions are sparse, selective, and robust. arXiv preprint, 2014.人脸识别上的进步香港中文大学DeepID2+在25个人脸Patch上分别训练CNN（4个卷积层，4个全连接层，4个verification损失信号和1个identification损失信号）训练数据：10K人，202K名人图像Y. Sun, X. Wang, and X. Tang, CVPR14人脸识别上的进步Google最新的FaceNet深层网络（22层）+ 海量数据(800万人，2亿张图像） + Triplet Loss (不需要额外占用显存)F. Schroff,

24、 D. Kalenichenko, and J. Philbin, CVPR15提纲深度学习(DL)及其应用前沿DL在CV领域应用的启示关键算法介绍BP算法Auto-EncoderCNNCNN主要变种关于DL的思考与讨论36DL之前的视觉处理方法分步处理背后的哲学分而治之Divide and Conquer Knowledge-drivenHand-crafted featureI think it should be solved by methods like 37DL及其之后的视觉处理方法38DL及其之后的视觉处理方法学习到接近期望的底层、中层和高层特征39DL之前的视觉处理方法任务人工

25、设计F（部分学习F）领域知识：分步处理滤波器，局部特征(SIFT)，BoW，直方图，Max/Sum汇聚，判别分析，Kernel技巧，分段线性，流形学习，测度学习40类标签(分类问题)向量(回归/估计)预处理特征设计特征降维分类/回归DL时代的视觉处理方法任务人工设计F（部分学习F）End-to-end地学习F（全步骤学习）Representation learningFeature learningNonlinear transform learning41离散类标签(分类问题)连续向量(回归/估计)Credit to Dr. Xiaogang WangDL时代的视觉处理方法42Collect

26、 dataPreprocessing 1Feature designClassifierEvaluationPreprocessing 2Collect dataFeature transformFeature transformClassifierDeep neural networkEvaluationvs.Credit to Dr. Xiaogang WangDL时代的视觉处理方法方法论上的变化从分治协同（joint）多步骤end-to-end learning更广义的检测与识别分割与识别43提纲深度学习(DL)及其应用前沿DL在CV领域应用的启示关键算法介绍Perceptron算法BP

27、算法Auto-EncoderCNN及其主要变种关于DL的思考与讨论44Perceptron45Frank Rosenblatt(1957), The Perceptron-a perceiving and recognizing automaton. Report 85-460-1, Cornell Aeronautical Laboratory.Perceptron算法46F. Rosenblatt. The perceptron: A probabilistic model for informationstorage and organization in the brain. Psyc

28、hological Review, 65:386-408, 1958Perceptron算法47前馈神经网络的BP学习算法David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). Learning representations by back-propagating errors. Nature323(6088): 533536（单独slides）卷积神经网络及其变种49（单独slides）提纲深度学习(DL)及其应用前沿DL在CV领域应用的启示关键算法介绍Perceptron及学习算法MLP及其BP算法Auto-EncoderCNN及其主要变种关于DL的思考与讨论50关于DL的更多讨论DL带来观念的变革DL是类脑信息处理方法吗？DL有理论吗？DL不能做什么？数据驱动的学习不再需要领域知识？工业界抢了学术界的饭碗？CV研究者沦为ML研究者的实验员？DL未来工作？51DL带来观念的变革人工领域知识驱动数据驱动的学习思想小数据控制模型复杂度避免过拟合大数据提高模型复杂度避免欠拟合“大数据+简单模型”是错误的！维数灾难(降维)高维有益(升维)分步、分治思想协同学习(joint learning)思想End-to-end的全过程学习软硬件更优的协