深度神经网络.ppt

深度神经网络IDeepNeuralNetworks,中国科学院自动化研究所吴高巍gaowei.wu2016-12-6,内容,深度神经网络发展历史、背景动机WhyDeepLearning?深度学习常用模型,NeuralnetworkBackpropagation,1986,解决了一般性学习问题与生物系统相联系,Nature,历史,NeuralnetworkBackpropagation,1986,Nature,历史,NeuralnetworkBackpropagation,1986,解决了一般性学习问题与生物系统相联系,Nature,历史,Butitisgivenup,SVMBoostingDecisiontree,2006,NeuralnetworkBackpropagation,1986,Nature,历史,2006,DeepbeliefnetScience,Unsupervised)概率值：p(v,h),p(v),p(h),p(v|h),p(h|v),RBM,极大似然,RBM,CD算法思想(Hinton,2002),ContrastiveDivergence,观测分布,真实分布,RBM,具体参数W,a,bRBM的能量：E(v,h)=vTWhbTvaTh概率形式,RBM,具体参数,RBM,对于hj0;1,vi0;1,可进一步化简：,RBM,通过采样来计算第二项：CD-K算法再回顾ML算法目标,RBM,算法流程(CD-1)：输入样本为v1,v2,vn，设观测变量v,隐变量h将对各参数的偏导数初始化为wij=0,aj=0,bi=0;Fork=1,N:v(0)vnForj=1,.,m,dosample：Fori=1,.,n,dosample：计算梯度，最后平均,（样本数）,（隐结点数）,（维数）,RBM,CD-1算法,RBM,图解,RBM,训练技巧（结构已定）将数据分成Batch,在每个batch内并行计算将CD-算法折衷成CD-1算法监控学习过程防止overfitting监控学习率增加动力机制（选样）增加稀疏机制（联接）,G.Hinton.APracticalGuidetoTrainingRestrictedBoltzmannMachines,TechReport,No.UTMLTR2010-003,DepartmentofComputerScience,UniversityofToronto,Canada,DeepBeliefNetworks(DBNs),概率生成模型深层结构多层非监督的预学习提供了网络好的初始化maximizingthelower-boundofthelog-likelihoodofthedata监督微调(fine-tuning)Generative:Up-downalgorithmDiscriminative:backpropagation,DBN结构,Hintonetal.,2006,DBNGreedytraining,第一步ConstructanRBMwithaninputlayervandahiddenlayerhTraintheRBM,DBNGreedytraining,第二步StackanotherhiddenlayerontopoftheRBMtoformanewRBMFixW1,sampleh1fromQ(h1|v)asinput.TrainW2asRBM.,DBNGreedytraining,第三步Continuetostacklayersontopofthenetwork,trainitaspreviousstep,withsamplesampledfromQ(h2|h1)Andsoon,Fine-tuning,Trainingforclassification,逐层贪婪训练,RBMspecifiesP(v,h)fromP(v|h)andP(h|v)ImplicitlydefinesP(v)andP(h)KeyideaofstackingKeepP(v|h)from1stRBMReplaceP(h)bythedistributiongeneratedby2ndlevelRBM,逐层贪婪训练,EasyapproximateinferenceP(hk+1|hk)approximatedfromtheassociatedRBMApproximationbecauseP(hk+1)differsbetweenRBMandDBNTraining:VariationalboundjustifiesgreedylayerwisetrainingofRBMs,DBN识别手写数字,待识别数字的黑白位图,隐性神经元,识别结果,关闭状态的神经元,开启状态的神经元,Resultisright,errorrate：DBN1.2%,ComparewithSVM1.4%,DBN提取音频特征,DBN训练一个5层的深度网络提取音乐的特征，用于音乐风格的分类,参考文献：PhilippeHamelandDouglasEck,LEARNINGFEATURESFROMMUSICAUDIOWITHDEEPBELIEFNETWORKS.,3个隐藏层,输入的原始数据是经过分帧，加窗之后的信号的频谱分类器采用的是支持向量机SVM对比的方法是MFCCs，分类器同样采用SVM,DBN提取音频特征,具有不同流派音频的不同表示的2维映射,和输入Inputs、MFCCs相比，DBN算法中隐藏层的激活的聚类表达更明确，分类的准确度更高。,小结,深度神经网络发展历史、背景动机WhyDeepLearning?深度学习常用模型,References,G.Hinton.APracticalGuidetoTrainingRestrictedBoltzmannMachines,TechReport,No.UTMLTR2010-003,DepartmentofComputerScience,UniversityofToronto,CanadaG.HintonandR.Salakhutdinov.ReducingtheDimensionalityofDatawithNeuralNetworks.Science,2006Vincent,Pascal,etal.StackedDenoisingAutoencoders:LearningUsefulRepresentationsinaDeepNetworkwithaLocalDenoisingCriterion.JournalofMa