台大-李宏毅-B站机器学习视频-课件神经网络与深度学习DL

DeepLearning,Deeplearningattractslotsofattention.,Ibelieveyouhaveseenlotsofexcitingresultsbefore.,DeeplearningtrendsatGoogle.Source:SIGMOD/JeffDean,UpsanddownsofDeepLearning,1958:Perceptron(linearmodel)1969:Perceptronhaslimitation1980s:Multi-layerperceptronDonothavesignificantdifferencefromDNNtoday1986:BackpropagationUsuallymorethan3hiddenlayersisnothelpful1989:1hiddenlayeris“goodenough”,whydeep?2006:RBMinitialization(breakthrough)2009:GPU2011:Starttobepopularinspeechrecognition2012:winILSVRCimagecompetition,ThreeStepsforDeepLearning,DeepLearningissosimple,NeuralNetwork,NeuralNetwork,“Neuron”,Differentconnectionleadstodifferentnetworkstructures,NeuralNetwork,Networkparameter:alltheweightsandbiasesinthe“neurons”,FullyConnectFeedforwardNetwork,1,-1,1,-2,1,-1,1,0,4,-2,0.98,0.12,FullyConnectFeedforwardNetwork,1,-2,1,-1,4,-2,0.98,0.12,2,-1,-1,-2,3,-1,4,-1,0.86,0.11,0.62,0.83,1,-1,FullyConnectFeedforwardNetwork,1,-2,1,-1,1,0,0.73,0.5,2,-1,-1,-2,3,-1,4,-1,0.72,0.12,0.51,0.85,0,0,-2,2,00=0.510.85,11=0.620.83,0,0,Thisisafunction.,Inputvector,outputvector,Givennetworkstructure,defineafunctionset,OutputLayer,HiddenLayers,InputLayer,FullyConnectFeedforwardNetwork,Input,Output,y1,y2,yM,neuron,8layers,19layers,22layers,AlexNet(2012),VGG(2014),GoogleNet(2014),16.4%,7.3%,6.7%,/slides/winter1516_lecture8.pdf,Deep=Manyhiddenlayers,AlexNet(2012),VGG(2014),GoogleNet(2014),152layers,3.57%,ResidualNet(2015),Taipei101,101layers,16.4%,7.3%,6.7%,Deep=Manyhiddenlayers,Specialstructure,MatrixOperation,1,-2,1,-1,1,0,4,-2,0.98,0.12,11,1211,+,10,0.980.12,=,1,-1,42,y1,y2,yM,NeuralNetwork,W1,W2,WL,b2,bL,x,a1,a2,y,aL-1,b1,=,y1,y2,yM,NeuralNetwork,W1,W2,WL,b2,bL,x,a1,a2,y,y,b1,W1,x,+,b2,W2,+,bL,WL,x,+,b1,Usingparallelcomputingtechniquestospeedupmatrixoperation,OutputLayer,y1,y2,yM,OutputLayer,HiddenLayers,InputLayer,Featureextractorreplacingfeatureengineering,=Multi-classClassifier,Softmax,ExampleApplication,Input,Output,16x16=256,Ink1Noink0,Eachdimensionrepresentstheconfidenceofadigit.,is1,is2,is0,0.1,0.7,0.2,Theimageis“2”,ExampleApplication,HandwritingDigitRecognition,Machine,“2”,is1,is2,is0,Whatisneededisafunction,Input:256-dimvector,output:10-dimvector,NeuralNetwork,OutputLayer,HiddenLayers,InputLayer,ExampleApplication,Input,Output,“2”,is1,is2,is0,AfunctionsetcontainingthecandidatesforHandwritingDigitRecognition,Youneedtodecidethenetworkstructuretoletagoodfunctioninyourfunctionset.,FAQ,Q:Howmanylayers?Howmanyneuronsforeachlayer?Q:Canthestructurebeautomaticallydetermined?E.g.EvolutionaryArtificialNeuralNetworksQ:Canwedesignthenetworkstructure?,ConvolutionalNeuralNetwork(CNN),ThreeStepsforDeepLearning,DeepLearningissosimple,NeuralNetwork,LossforanExample,y1,y2,y10,CrossEntropy,“1”,1,0,0,target,Softmax,=110,1,2,10,Givenasetofparameters,TotalLoss,NN,NN,NN,1,2,1,NN,3,Foralltrainingdata,=1,FindthenetworkparametersthatminimizetotallossL,TotalLoss:,2,3,FindafunctioninfunctionsetthatminimizestotallossL,ThreeStepsforDeepLearning,DeepLearningissosimple,NeuralNetwork,GradientDescent,1,Compute1,1,0.15,2,Compute2,2,0.05,1,Compute1,1,0.2,0.2,-0.1,0.3,121,=,gradient,GradientDescent,1,Compute1,1,0.15,1,Compute1,0.09,2,Compute2,2,0.05,2,Compute2,0.15,1,Compute1,1,0.2,1,Compute1,0.10,0.2,-0.1,0.3,GradientDescent,Thisisthe“learning”ofmachinesindeeplearning,Evenalphagousingthisapproach.,Ihopeyouarenottoodisappointed:p,Peopleimage,Actually.,Backpropagation,Backpropagation:anefficientwaytocomputeinneuralnetwork,Ref:.tw/tlkagk/courses/MLDS_2015_2/Lecture/DNN%20backprop.ecm.mp4/index.html,ConcludingRemarks,NeuralNetwork,Whatarethebenefitsofdeeparchitecture?,DeeperisBetter?,Seide,Frank,GangLi,andDongYu.ConversationalSpeechTranscriptionUsingContext-DependentDeepNeuralNetworks.Interspeech.2011.,Notsurprised,moreparameters,betterperformance,UniversalityTheorem,Referenceforthereason:,Anycontinuousfunctionf,Canberealizedbyanetworkwithonehiddenlayer,(givenenough