21-循环神经网络

recurrent

neural

networkDeepLearninganditsApplicationWhydoWeNeedRNN?DNNstructureisgreatforinputoffixedsize.SJTUDeepLearningLecture.2Q:Whatifwewanttomakepredictionfromasequenceofevents?。。。Today’sweather？SJTUDeepLearningLecture.3WhydoWeNeedRNN?Solution1:Onlyconsiderthemostrecentseveraldays:。。。Today’sweather？Solution2:Defineawaythat“encodes”thewholehistory.RNNdoesthatinaneuralnetworkstyle.SJTUDeepLearningLecture.4WhydoWeNeedRNN?SerialorderMichaelJordan,Serialorder:Aparalleldistributedprocessingapproach(Tech.Rep.No.8604).SanDiego:UniversityofCalifornia,InstituteforCognitiveScience.“Itismyviewthatmanyoftheseproblemsdisappearwhenacleardistinctionismadebetweenthestateofthesystemandtheoutputofthesystem.“-MichaelI.JordanSerialorder:aparalleldistributedprocessingapproach.Technicalreport,June1985-March1986ElmannetworkJeffreyElman,FindingStructureinTime,1990EarlyStudy(Elmannetwork)SJTUDeepLearningLecture.[1]MichaelJordan,Serialorder:Aparalleldistributedprocessingapproach(Tech.Rep.No.8604).SanDiego:UniversityofCalifornia,InstituteforCognitiveScience.[2]JeffreyElman,FindingStructureinTime,1990SerialorderElmannetwork(trainedusingbackpropagation)outputtostatestatetostateSerialorder

(remake

version)Dashedline:fromtimet-1totCalledstateunitsinJordan’spaperMichaelJordan,Serialorder:Aparalleldistributedprocessingapproach(Tech.Rep.No.8604).SanDiego:UniversityofCalifornia,InstituteforCognitiveScience.Elmannetwork(remake

version)Dashedline:fromtimet-1totJeffreyElman,FindingStructureinTime,1990RevisitXORProblemHowtomakeitatemporallearningtask?Ellman’sway:XY->Z[XYZ]XYistheinputtwobitsZistheoutputbitZ=XOR(X,Y)0,01,01,10,1100TheresultsofXORRNNtestRecurrentNeuralNetwork(RNN)SJTUDeepLearningLecture.12UnfoldingRNN(compactversion)SJTUDeepLearningLecture.13HowdoWeUseRNNUseitgeneratively:Useitdiscriminatively:cloudysunnycloudyrainyrainysunnyNounIlikeVerbstudyingNounSJTUDeepLearningLecture.14DeepRNNWecanstackmultiplerecurrentlayerstogeta“deeprecurrentnetwork”.SJTUDeepLearningLecture.15Fromtimet+1totFromlayerl+1tol

VariantsofdeeperRNNPascanuetal,HowtoConstructDeepRecurrentNeuralNetworks,arXiv:1312.6026TypicalTasksbyRNNObjectiveFunctionforRNNL1L2L3LTUsuallywemakepredictionateverytimestamp.Wrec,WinRememberthatparametersaresharedforeachtimestep.SJTUDeepLearningLecture.18Back-propagationThroughTime(BPTT)Let'sconsideronlywithT=2AndcalculateerrorvectorsintheNN.SJTUDeepLearningLecture.19Back-propagationThroughTime(BPTT)Afterwegettheerrors,weusethemtocalculategradients,forexample:SJTUDeepLearningLecture.20BPTTQuestion:dowealwaysneedtoback-propagatetothebeginning?Answer:Intheory,yes,inpractice,weusetruncatedBPTT.Notethatinmostcases,thedataconsistsofindependentshortsequence.SJTUDeepLearningLecture.TruncatedBPTTTruncatedBPTTmeansthatweonlyback-propagatethroughafewfixedtimestamps.Thisisanapproximationoffullback-propagationthroughtime.L(0)L(1)L(2)L(3)L(4)L(5)L(6)L(7)…SJTUDeepLearningLecture.22BPTT:WasteofComputationMindthatifweuseBPTTforeachtimestamp,alotofback-propagationarerepeated.E.g.considererroratz5L(0)L(1)L(2)L(3)L(4)L(5)L(6)L(7)…repeated.SJTUDeepLearningLecture.23Combine!BPTTBlockInthiscase,errorforz(6)onlyneedstobeback-propagatedonce.Wecansimplyaccumulatetheerrorswithrespecttotailingtargets.Inpractice,weusuallysetthebptt-blockto~15,andbpttto~5SJTUDeepLearningLecture.24DifficultyofTrainingRNNInback-propagationofRNN,therearetwoproblemscalled:vanishinggradientandexplodinggradient.SJTUDeepLearningLecture.25Exploding/VanishingGradientWhenthedeterminantofWrecaresmall,gradientvanishes(asinmostcases),whenthey’relarge(rarely),thegradientwillexplode.SJTUDeepLearningLecture.26Trick:GradientClippingTodealwithexplodinggradient.Gradientclippingisneededtomaketrainingstable.If