台大-李宏毅-B站机器学习视频-课件神经网络与深度学习RNN training

TrainingRecurrentNeuralNetwork,Hung-yiLee,Goal,1,2,3,0,AllelementinWh,WiorWo,=12=12,=2,Backpropagationthroughtime(BPTT),Review:Backpropagation,ForwardPass,BackwardPass,Errorsignal,Review:Backpropagation,LayerL,LayerL-1,BackwardPass,Errorsignal,BackpropagationthroughTime,yn,Averydeepneuralnetwork,output:yn,Input:init,x1,x2,xn,target:,UNFOLD:,an-1,x1,a1,init,an-2,BackpropagationthroughTime,yn,Averydeepneuralnetwork,output:yn,Input:init,x1,x2,xn,target:,UNFOLD:,an-1,x1,a1,init,an-2,BackpropagationthroughTime,yn,Averydeepneuralnetwork,output:yn,Input:init,x1,x2,xn,target:,UNFOLD:,an-1,x1,a1,init,an-2,yn,an-1,x1,a1,Someweightsareshared.,init,an-2,j,i,k,i,k,j,i,j,i,k,j,111,222,2,1,1,2,BackpropagationthroughTime,Averydeepneuralnetwork,output:yn,Input:init,x1,x2,xn,target:,UNFOLD:,Initializew1,w2bythesamevalue,(Thevaluesofw1,w2shouldalwaysbethesame.),thesamememory,pointer,pointer,BPTT,1,2,3,For1,4,BackwardPass:,For2,For3,For4,1,2,3,4,ForwardPass:,Computea1,a2,a3,a4,a1,a2,a3,a4,Unfortunately,itisnoteasytotrainRNN.,Theerrorsurfaceisrough.,w1,w2,Cost,Source:/proceedings/papers/v28/pascanu13.pdf,Theerrorsurfaceiseitherveryflatorverysteep.,Ifn=1000:,ToyExample,1,0,1,w,y1,0,1,w,y2,0,1,w,y3,0,1,w,yn,CostCn,=,=1,=1.01,=1,20000,=0.99,=0.01,0,0,1,1,1,1,n=10,n=100,n=1000,Onlyextremelylargeandsmallvalue,=1,=,BackpropagationthroughTime,yn,an-1,x1,a1,init,an-2,GradientVanishing/Exploding,Forsimplicity,assumelinearactivationfunction,1=,1,2,1,2=2,=,1=1,GradientVanishing/Exploding,1step,2steps,5steps,10steps,20steps,50steps,PossibleSolutions,ClippedGradient,w1,w2,Cost,Clippedgradient,theano.tensor.clip(x,min,max),Source:/proceedings/papers/v28/pascanu13.pdf,Source:/fritz/absps/momentum.pdf,Gradientdescent,Momentum,NesterovsAcceleratedGradient(NAG),NAG,Methods:,Valley,NAG,Momentum,NesterovsAcceleratedGradient(NAG),Movement,Gradient,LastMovement,Gradient=0,Gradient=0,RMSProp,LargerLearningRate,SmallerLearningRate,Review:Adagrad,+1=02,Usefirstderivativetoestimatesecondderivative,RMSProp,1,2,ErrorSurfacecanbeevenmorecomplexwhentrainingRNN.,LargerLearningRate,SmallerLearningRate,RMSProp,1000,2111,+1,0=0,1=02+112,3222,2=12+122,=12+12,RootMeanSquareofthegradientswithpreviousgradientsbeingdecayed,x1,x2,+,+,+,+,+,+,+,+,Input,1,2,4timesofparameters,LSTMcanaddressthegradientvanishingproblem.,LSTM,z,zi,zf,zo,yt,1,z,zi,zf,zo,Extension:“peephole”,+1,LSTM,z,zi,zf,zo,yt,1,z,zi,zf,zo,+1,a,b,a,b,1,1,XWT,XWT,ConstantErrorCarrousel(CEC),OtherSimplerVariants,GRU:Cho,Kyunghyun,etal.LearningPhraseRepresentationsusingRNNEncoderDecoderforStatisticalMachineTranslation“,EMNLP,2014SCRN:Mikolov,Tomas,etal.Learninglongermemoryinrecurrentneuralnetworks“,ICLR2015,BetterInitialization,VanillaRNN:InitializedwithIdentitymatrix+ReLU,QuocV.Le,NavdeepJaitly,GeoffreyE.Hinton,“ASimpleWaytoInitializeRecurrentNetworksofRectifiedLinearUnits“,2015,ConcludingRemarks,Be