资料课件讲义RNN

RecurrentNeuralNetwork(RNN),ExampleApplication,SlotFilling,IwouldliketoarriveTaipeionNovember2nd.,ticketbookingsystem,Destination:,timeofarrival:,Taipei,November2nd,ExampleApplication,Taipei,Input:aword,(Eachwordisrepresentedasavector),SolvingslotfillingbyFeedforwardnetwork?,1-of-Nencoding,Eachdimensioncorrespondstoawordinthelexicon,Thedimensionforthewordis1,andothersare0,lexicon=apple,bag,cat,dog,elephant,apple=10000,bag=01000,cat=00100,dog=00010,elephant=00001,Thevectorislexiconsize.,1-of-NEncoding,Howtorepresenteachwordasavector?,Beyond1-of-Nencoding,w=“apple”,a-a-a,a-a-b,p-p-l,26X26X26,a-p-p,p-l-e,1,1,1,0,0,Wordhashing,Dimensionfor“Other”,w=“Sauron”,apple,bag,cat,dog,elephant,“other”,0,0,0,0,0,1,w=“Gandalf”,5,ExampleApplication,Taipei,dest,timeofdeparture,Input:aword,(Eachwordisrepresentedasavector),Output:,Probabilitydistributionthattheinputwordbelongingtotheslots,SolvingslotfillingbyFeedforwardnetwork?,ExampleApplication,Taipei,arriveTaipeionNovember2nd,other,other,dest,time,time,leaveTaipeionNovember2nd,placeofdeparture,Neuralnetworkneedsmemory!,dest,timeofdeparture,Problem?,RecurrentNeuralNetwork(RNN),Memorycanbeconsideredasanotherinput.,Theoutputofhiddenlayerarestoredinthememory.,store,Example,store,Alltheweightsare“1”,nobias,Allactivationfunctionsarelinear,1,1,0,0,givenInitialvalues,2,2,4,4,outputsequence:,44,Example,store,Alltheweightsare“1”,nobias,Allactivationfunctionsarelinear,1,1,2,2,6,6,12,12,outputsequence:,44,1212,Example,store,Alltheweightsare“1”,nobias,Allactivationfunctionsarelinear,2,2,6,6,16,16,32,32,outputsequence:,44,1212,3232,Changingthesequenceorderwillchangetheoutput.,RNN,store,store,x1,x2,x3,y1,y2,y3,a1,a1,a2,a2,a3,Thesamenetworkisusedagainandagain.,arriveTaipeionNovember2nd,Probabilityof“arrive”ineachslot,Probabilityof“Taipei”ineachslot,Probabilityof“on”ineachslot,RNN,leave,Taipei,Probof“leave”ineachslot,Probof“Taipei”ineachslot,Probof“arrive”ineachslot,Probof“Taipei”ineachslot,arrive,Taipei,Different,Thevaluesstoredinthememoryisdifferent.,Ofcourseitcanbedeep,xt,xt+1,xt+2,yt,yt+1,yt+2,ElmanNetwork&JordanNetwork,xt,xt+1,yt,yt+1,Wh,Wi,Wo,Wi,Wo,xt,xt+1,yt,yt+1,Wh,Wi,Wo,Wi,Wo,ElmanNetwork,JordanNetwork,BidirectionalRNN,yt+1,yt+2,yt,xt,xt+1,xt+2,xt,xt+1,xt+2,MemoryCell,LongShort-termMemory(LSTM),InputGate,OutputGate,Signalcontroltheinputgate,Signalcontroltheoutputgate,ForgetGate,Signalcontroltheforgetgate,Otherpartofthenetwork,Otherpartofthenetwork,(Otherpartofthenetwork),(Otherpartofthenetwork),(Otherpartofthenetwork),LSTM,SpecialNeuron:4inputs,1output,multiply,multiply,Activationfunctionfisusuallyasigmoidfunction,Between0and1,Mimicopenandclosegate,c,=+,=,LSTM-Example,100,0,1,2,3,310,0,200,0,410,0,200,0,101,7,3-10,0,610,0,101,6,Whenx2=1,addthenumbersofx1intothememory,Whenx3=1,outputthenumberinthememory.,0,0,3,3,7,7,7,0,6,Whenx2=-1,resetthememory,1,2,3,310,0,410,0,200,0,101,7,3-10,0,x1,x2,x3,x1,x2,x3,x1,x2,x3,x1,x2,x3,y,1,0,0,100,0,0,1,1,1,1,0,-10,0,0,100,-10,0,0,10,100,0,1,2,3,310,0,410,0,200,0,101,7,3-10,0,3,1,0,3,1,0,3,1,0,3,1,0,y,1,0,0,100,0,0,1,1,1,1,0,-10,0,0,100,-10,0,0,10,100,3,1,3,1,0,3,3,0,0,1,2,3,310,0,410,0,200,0,101,7,3-10,0,4,1,0,4,1,0,4,1,0,4,1,0,y,1,1,1,1,4,1,4,1,3,7,7,0,0,1,0,0,100,0,0,0,-10,0,0,100,-10,0,0,10,100,1,2,3,310,0,410,0,200,0,101,7,3-10,0,2,0,0,2,0,0,2,0,0,2,0,0,y,1,1,1,1,2,0,0,1,7,7,0,0,1,0,0,100,0,0,0,-10,0,0,100,-10,0,0,10,100,1,2,3,310,0,410,0,200,0,101,7,3-10,0,1,0,1,1,0,1,1,0,1,1,0,1,y,1,1,1,1,1,0,0,1,7,7,1,7,1,0,0,100,0,0,0,-10,0,0,100,-10,0,0,10,100,1,2,3,310,0,410,0,200,0,101,7,3-10,0,3,-1,0,3,-1,0,3,-1,0,3,-1,0,y,1,1,1,1,3,0,0,0,7,0,0,0,0,1,0,0,100,0,0,0,-10,0,0,100,-10,0,0,10,100,x1,x2,Input,OriginalNetwork:,SimplyreplacetheneuronswithLSTM,1,2,1,2,x1,x2,+,+,+,+,+,+,+,+,Input,1,2,4timesofparameters,LSTM,vector,z,zi,zf,zo,4vectors,LSTM,z,zi,zf,zo,yt,z,zi,zf,zo,LSTM,z,zi,zf,zo,yt,z,zi,zf,zo,Extension:“peephole”,Multiple-layerLSTM,Thisisquitestandardnow.,/2015-09-20/src/14426967627131.gif,Dontworryifyoucannotunderstandthis.Kerascanhandleit.,Kerassupports“LSTM”,“GRU”,“SimpleRNN”layers,copy,copy,x1,x2,x3,y1,y2,y3,Wi,a1,a1,a2,a2,a3,arriveTaipeionNovember2nd,TrainingSentences:,LearningTarget,other,other,dest,other,dest,other,time,time,Learning,Backpropagationthroughtime(BPTT),copy,Unfortunately,RNN-basednetworkisnotalwayseasytolearn,感謝曾柏翔同學提供實驗結果,RealexperimentsonLanguagemodeling,Lucky,sometimes,TotalLoss,Epoch,Theerrorsurfaceisrough.,w1,w2,Cost,Theerrorsurfaceiseitherveryflatorverysteep.,Clipping,RazvanPascanu,ICML13,TotalLoss,Why?,1,1,y1,0,1,w,y2,0,1,w,y3,0,1,w,y1000,=1,=1.01,1000=1,100020000,=0.99,=0.01,10000,10000,1,1,1,1,Large,SmallLearningrate?,small,LargeLearningrate?,ToyExample,=w999,add,LongShort-termMemory(LSTM)Candealwithgradientvanishing(notgradientexplode),HelpfulTechniques,Memoryandinputareadded,Theinfluenceneverdisappearsunlessforgetgateisclosed,NoGradientvanishing,(Ifforgetgateisopened.),Cho,EMNLP14,GatedRecurrentUnit(GRU):simplerthanLSTM,HelpfulTechniques,VanillaRNNInitializedwithIdentitymatrix+ReLUactivationfunctionQuocV.Le,arXiv15,OutperformorbecomparablewithLSTMin4differenttasks,JanKoutnik,JMLR14,ClockwiseRNN,TomasMikolov,ICLR15,StructurallyConstrainedRecurrentNetwork(SCRN),MoreApplications,store,store,x1,x2,x3,y1,y2,y3,a1,a1,a2,a2,a3,arriveTaipeionNovember2nd,Probabilityof“arrive”ineachslot,Probabilityof“Taipei”ineachslot,Probabilityof“on”ineachslot,Inputandoutputarebothsequenceswiththesamelength,RNNcandomorethanthat!,Manytoone,Inputisavectorsequence,butoutputisonlyonevector,SentimentAnalysis,我,覺,太,得,糟,了,超好雷,好雷,普雷,負雷,超負雷,看了這部電影覺得很高興.,這部電影太糟了.,這部電影很棒.,Positive(正雷),Negative(負雷),Positive(正雷),Manytoone,Inputisavectorsequence,butoutputisonlyonevector,KeyTermExtraction,Shen&Lee,Interspeech16,iVi,document,EmbeddingLayer,KeyTerms:DNN,LSTN,ManytoMany(Outputisshorter),Bothinputandoutputarebothsequences,buttheoutputisshorter.E.g.SpeechRecognition,好,好,好,Trimming,棒,棒,棒,棒,棒,“好棒”,Whycantitbe“好棒棒”,Input:,Output:,(charactersequence),(vectorsequence),Problem?,ManytoMany(Outputisshorter),Bothinputandoutputarebothsequences,buttheoutputisshorter.ConnectionistTemporalClassification(CTC)AlexGraves,ICML06AlexGraves,ICML14HaimSak,Interspeech15JieLi,Interspeech15AndrewSenior,ASRU15,“好棒”,“好棒棒”,Addanextrasymbol“”representing“null”,ManytoMany(Outputisshorter),CTC:Training,AcousticFeatures:,Label:,Allpossiblealignmentsareconsideredascorrect.,ManytoMany(Outputisshorter),CTC:example,Graves,Alex,andNavdeepJaitly.Towardsend-to-endspeechrecognitionwithrecurrentneuralnetworks.Proceedingsofthe31stInternationalConferenceonMachineLearning(ICML-14).2014.,HISFRIENDS,ManytoMany(NoLimitation),Bothinputandoutputarebothsequenceswithdifferentlengths.SequencetosequencelearningE.g.MachineTranslation(machinelearning機器學習),Containingallinformationaboutinputsequence,ManytoMany(NoLimitation),Bothinputandoutputarebothsequenceswithdifferentlengths.SequencetosequencelearningE.g.MachineTranslation(machinelearning機器學習),機,習,Dontknowwhentostop,慣,性,ManytoMany(NoLimitation),推tlkagk:=斷=,接龍推文是ptt在推文中的一種趣味玩法，與推齊有些類似但又有所不同，是指在推文中接續上一樓的字句，而推出連續的意思。該類玩法確切起源已不可知(鄉民百科),ManytoMany(NoLimitation),Bothinputandoutputarebothsequenceswithdifferentlengths.SequencetosequencelearningE.g.MachineTranslation(machinelearning機器學習),機,習,Addasymbol“=“(斷),IlyaSutskever,NIPS14DzmitryBahdanau,arXiv15,=,ManytoMany(NoLimitation),Bothinputandoutputarebothsequenceswithdifferentlengths.SequencetosequencelearningE.g.MachineTranslation(machinelearning機器學習),/pdf/1612.01744v1.pdf,BeyondSequence,Syntacticparsing,OriolVinyals,LukaszKaiser,TerryKoo,SlavPetrov,IlyaSutskever,GeoffreyHinton,GrammarasaForeignLanguage,NIPS2015,john,has,a,dog,Sequence-to-sequenceAuto-encoder-Text,Tounderstandthemeaningofawordsequence,theorderofthewordscannotbeignored.,whitebloodcellsdestroyinganinfection,aninfectiondestroyingwhitebloodcells,exactlythesamebag-of-word,positive,negative,differentmeaning,Sequence-to-sequenceAuto-encoder-Text,Li,Jiwei,Minh-ThangLuong,andDanJurafsky.Ahierarchicalneuralautoencoderforparagraphsanddocuments.arXivpreprintarXiv:1506.01057(2015).,Sequence-to-sequenceAuto-encoder-Text,Li,Jiwei,Minh-ThangLuong,andDanJurafsky.Ahierarchicalneuralautoencoderforparagraphsanddocuments.arXivpreprintarXiv:1506.01057(2015).,Sequence-to-sequenceAuto-encoder-Speech,Dimensionreductionforasequencewithvariablelength,ever,ever,never,never,never,dog,dog,dogs,Fixed-lengthvector,audiosegments(word-level),Yu-AnChung,Chao-ChungWu,Chia-HaoShen,Hung-YiLee,Lin-ShanLee,AudioWord2Vec:UnsupervisedLearningofAudioSegmentRepresentationsusingSequence-to-sequenceAutoencoder,Interspeech2016,Sequence-to-sequenceAuto-encoder-Speech,Audioarchivedividedintovariable-lengthaudiosegments,AudioSegmenttoVector,AudioSegmenttoVector,Similarity,SearchResult,SpokenQuery,Off-line,On-line,Sequence-to-sequenceAuto-encoder-Speech,audiosegment,acousticfeatures,Thevaluesinthememoryrepresentthewholeaudiosegment,x1,x2,x3,x4,RNNEncoder,audiosegment,vector,Thevectorwewant,HowtotrainRNNEncoder?,Sequence-to-sequenceAuto-encoder,RNNDecoder,x1,x2,x3,x4,y1,y2,y3,y4,x1,x2,x3,x4,RNNEncoder,audiosegment,acousticfeatures,TheRNNencoderanddecoderarejointlytrained.,Inputacousticfeatures,Sequence-to-sequenceAuto-encoder-Speech,Visualizingembeddingvectorsofthewords,fear,near,name,fame,Demo:Chat-bot,電視影集(40,000sentences)、美國總統大選辯論,Demo:Chat-bot,DevelopTeamInterfacedesign:Prof.Lin-LinChen&ArronLuWebprogramming:Shi-YunHuangDatacollection:Chao-ChuangShihSystemimplementation:KevinWu,DerekChuang,&Zhi-WeiLee(李致緯),RoyLu(盧柏儒)Systemdesign:RichardTsai&Hung-YiLee,61,Demo:VideoCaptionGeneration,Video,Agirlisrunning.,Agroupofpeopleiswalkingintheforest.,Agroupofpeopleisknockedbyatree.,Demo:VideoCaptionGeneration,Canmachinedescribewhatitseefromvideo?Demo:台大語音處理實驗室曾柏翔、吳柏瑜、盧宏宗Video:莊舜博、楊棋宇、黃邦齊、萬家宏,Demo:ImageCaptionGeneration,Inputanimage,butoutputasequenceofwords,Inputimage,a,woman,is,=,CNN,Avectorforwholeimage,KelvinXu,arXiv15LiYao,ICCV15,CaptionGeneration,Demo:ImageCaptionGeneration,Canmachinedescribewhatitseefromimage?Demo:台大電機系大四蘇子睿、林奕辰、徐翊祥、陳奕安,MTK產學大聯盟,.tw/photo/politics/breakingnews/975542_1,Organize,Attention-basedModel,http:/henrylo1605.blogspot.tw/2015/05/blog-post_56.html,Breakfasttoday,Whatyoulearnedintheselectures,summervacation10yearsago,Whatisdeeplearning?,Answer,Attention-basedModel,ReadingHeadController,Input,ReadingHead,output,DNN/RNN,Ref:.tw/tlkagk/courses/MLDS_2015_2/Lecture/Attain%20(v3).ecm.mp4/index.html,Attention-basedModelv2,ReadingHeadController,Input,ReadingHead,output,DNN/RNN,NeuralTuringMachine,WritingHeadController,WritingHead,ReadingComprehension,Query,Eachsentencebecomesavector.,DNN/RNN,ReadingHeadController,answer,SemanticAnalysis,ReadingComprehension,End-To-EndMemoryNetworks.S.Sukhbaatar,A.Szlam,J.Weston,R.Fergus.NIPS,2015.,Thepositionofreadinghead:,Kerashasexample:,VisualQuestionAnswering,source:/,VisualQuestionAnswering,Query,DNN/RNN,ReadingHeadController,answer,CNN,Avectorforeachregion,SpeechQuestionAnswering,TOEFLListeningComprehensionTestbyMachineExample:,Question:“WhatisapossibleoriginofVenusclouds?”,AudioStory:,Choices:,(A)gasesreleasedasaresultofvolcanicactivity,(B)chemicalreactionscausedbyhighsurfacetemperatures,(C)burstsofradioenergyfromtheplanessurface,(D)strongwindsthatblowdustintotheatmosphere,(Theoriginalstoryis5minlong.),ModelArchitecture,QuestionSemantics,Itbequitepossiblethatthisbeduetovolcaniceruptionbecausevolcaniceruptionoftenemitgas.IfthatbethecasevolcanismcouldverywellbetherootcauseofVenussthickcloudcover.Andalsowehaveobserveburstofradioenergyfromtheplanetssurface.Theseburstbesimilartowhatweseewhenvolcanoeruptonearth,AudioStory:,SpeechRecognition,SemanticAnalysis,SemanticAnalysis,Attention,Answer,Selectthechoicemostsimilartotheanswer,Attention,Everythingislearnedfromtrainingexamples,SimpleBaselines,Accuracy(%),(1),(2),(3),(4),(5),(6),(7),NaiveApproaches,random,(4)thechoicewithsemanticmostsimilartoothers,(2)selecttheshortestchoiceasanswer,Experimentalsetup:717fortraining,124forvalidation,122fortesting,MemoryNetwork,Accuracy(%),(1),(2),(3),(4),(5),(6),(7),MemoryNetwork:39.2%,NaiveApproaches,(proposedbyFBAIgroup),ProposedApproach,Accuracy(%),(1),(2),(3),(4),(5),(6),(7),MemoryNetwork:39.2%,NaiveApproaches,ProposedApproach:48.8%,(proposedbyFBAIgroup),Fang&Hsu&Lee,SLT16,Tseng&Lee,Interspeech16,ToLearnMore,TheUnreasonableEffectivenessofRecurrentNeuralNetworkshttp:/karpathy.github.io/2015/05/21/rnn-effectiveness/UnderstandingLSTMNetworkshttp:/colah.github.io/posts/2015-08-Understanding-LSTMs/,Deep&Structured,RNNv.s.StructuredLearning,RNN,LSTMUnidirectionalRNNdoesnotconsiderthewholesequenceCostanderrornotalwaysrelatedDeep,HMM,CRF,StructuredPerceptron/SVMUsingViterbi,soconsiderthewholesequenceHowaboutBidirectionalRNN?CanexplicitlyconsiderthelabeldependencyCostistheupperboundoferror,?,IntegratedTogether,RNN,LSTM,HMM,CRF,StructuredPerceptron/