台大-李宏毅-B站机器学习视频-课件神经网络与深度学习transfer

TransferLearning,TransferLearning,Dog/CatClassifier,cat,dog,Datanotdirectlyrelatedtothetaskconsidered,elephant,tiger,Similardomain,differenttasks,Differentdomains,sametask,dog,cat,Why?,TaskConsidered,ImageRecognition,Datanotdirectlyrelated,SpeechRecognition,Specificdomain,English,Chinese,Webpages,MedicalImages,http:/www.bigr.nl/website/structure/main.php?page=researchlines&subpage=project&id=64,TextAnalysis,.hk/Translation-company-Directory.html,TransferLearning,Exampleinreallife,爆漫王,研究生生存守則,責編,漫畫家,投稿jump,畫分鏡,指導教授,研究生,投稿期刊,跑實驗,漫畫家on-line,研究生on-line,(wordembeddingknowsthat),TransferLearning-Overview,TargetData,SourceData(notdirectlyrelatedtothetask),labelled,labelled,unlabeled,unlabeled,ModelFine-tuning,Warning:differentterminologyindifferentliterature,ModelFine-tuning,TaskdescriptionTargetdata:,Sourcedata:,Example:(supervised)speakeradaptionTargetdata:audiodataanditstranscriptionsofspecificuserSourcedata:audiodataandtranscriptionsfrommanyspeakersIdea:trainingamodelbysourcedata,thenfine-tunethemodelbytargetdataChallenge:onlylimitedtargetdata,sobecarefulaboutoverfitting,Verylittle,Alargeamount,One-shotlearning:onlyafewexamplesintargetdomain,ConservativeTraining,Sourcedata(e.g.AudiodataofManyspeakers),Targetdata(e.g.Alittledatafromtargetspeaker),Inputlayer,Outputlayer,initialization,Inputlayer,Outputlayer,outputclose,parameterclose,LayerTransfer,Inputlayer,Outputlayer,Copysomeparameters,Targetdata,Sourcedata,1.Onlytraintherestlayers(preventoverfitting),2.fine-tunethewholenetwork(ifthereissufficientdata),LayerTransfer,Whichlayercanbetransferred(copied)?Speech:usuallycopythelastfewlayersImage:usuallycopythefirstfewlayers,LayerTransfer-Image,JasonYosinski,JeffClune,YoshuaBengio,HodLipson,“Howtransferablearefeaturesindeepneuralnetworks?”,NIPS,2014,Onlytraintherestlayers,fine-tunethewholenetwork,Source:500classesfromImageNet,Target:another500classesfromImageNet,LayerTransfer-Image,JasonYosinski,JeffClune,YoshuaBengio,HodLipson,“Howtransferablearefeaturesindeepneuralnetworks?”,NIPS,2014,Onlytraintherestlayers,TransferLearning-Overview,TargetData,SourceData(notdirectlyrelatedtothetask),labelled,labelled,unlabeled,unlabeled,MultitaskLearning,Fine-tuning,MultitaskLearning,Themulti-layerstructuremakesNNsuitableformultitasklearning,Inputfeature,TaskA,TaskB,TaskA,TaskB,InputfeaturefortaskA,InputfeaturefortaskB,MultitaskLearning-MultilingualSpeechRecognition,acousticfeatures,statesofFrench,statesofGerman,Humanlanguagessharesomecommoncharacteristics.,statesofSpanish,statesofItalian,statesofMandarin,Similarideaintranslation:DaxiangDong,HuaWu,WeiHe,DianhaiYuandHaifengWang,Multi-tasklearningformultiplelanguagetranslation.“,ACL2015,MultitaskLearning-Multilingual,Huang,Jui-Ting,etal.Cross-languageknowledgetransferusingmultilingualdeepneuralnetworkwithsharedhiddenlayers.ICASSP,2013,CharacterErrorRate,HoursoftrainingdataforMandarin,Mandarinonly,WithEuropeanLanguage,ProgressiveNeuralNetworks,input,Task1,input,Task2,input,Task3,AndreiA.Rusu,NeilC.Rabinowitz,GuillaumeDesjardins,HubertSoyer,JamesKirkpatrick,KorayKavukcuoglu,RazvanPascanu,RaiaHadsell,“ProgressiveNeuralNetworks”,arXivpreprint2016,TransferLearning-Overview,TargetData,SourceData(notdirectlyrelatedtothetask),labelled,labelled,unlabeled,unlabeled,MultitaskLearning,Fine-tuning,Domain-adversarialtraining,Taskdescription,Sourcedata:,Targetdata:,Trainingdata,Testingdata,mismatch,withlabel,withoutlabel,Domain-adversarialtraining,Domain-adversarialtraining,SimilartoGAN,Tooeasytofeatureextractor,featureextractor,Domainclassifier,Domain-adversarialtraining,Notonlycheatthedomainclassifier,butsatisfyinglabelclassifieratthesametime,Thisisabignetwork,butdifferentpartshavedifferentgoals.,Maximizelabelclassificationaccuracy,Maximizedomainclassificationaccuracy,Maximizelabelclassificationaccuracy+minimizedomainclassificationaccuracy,featureextractor,Domainclassifier,Labelpredictor,Domain-adversarialtraining,YaroslavGanin,VictorLempitsky,UnsupervisedDomainAdaptationbyBackpropagation,ICML,2015,HanaAjakan,PascalGermain,HugoLarochelle,FranoisLaviolette,MarioMarchand,Domain-AdversarialTrainingofNeuralNetworks,JMLR,2016,Domainclassifierfailsintheend,Itshouldstruggle,Domain-adversarialtraining,YaroslavGanin,VictorLempitsky,UnsupervisedDomainAdaptationbyBackpropagation,ICML,2015,HanaAjakan,PascalGermain,HugoLarochelle,FranoisLaviolette,MarioMarchand,Domain-AdversarialTrainingofNeuralNetworks,JMLR,2016,TransferLearning-Overview,TargetData,SourceData(notdirectlyrelatedtothetask),labelled,labelled,unlabeled,unlabeled,MultitaskLearning,Fine-tuning,Domain-adversarialtraining,Zero-shotlearning,Zero-shotLearning,Sourcedata:,Targetdata:,Differenttasks,Inspeechrecognition,wecannothaveallpossiblewordsinthesource(training)data.,Trainingdata,Testingdata,cat,dog,Howwesolvethisprobleminspeechrecognition?,:,:,:,Zero-shotLearning,Representingeachclassbyitsattributes,sufficientattributesforonetoonemapping,class,attributes,NN,Training,NN,furry,4legs,tail,furry,4legs,tail,1,1,1,1,0,0,Database,Zero-shotLearning,Representingeachclassbyitsattributes,class,attributes,NN,Attributes,Findtheclasswiththemostsimilarattributes,Testing,sufficientattributesforonetoonemapping,Zero-shotLearning,Attributeembedding,EmbeddingSpace,y2(attributeofdog),x1,x2,x3,y1(attributeofchimp),y3(attributeofGrass-mudhorse),1,2,3,2,andgcanbeNN.,Trainingtarget:,andascloseaspossible,3,1,Zero-shotLearning,Attributeembedding+wordembedding,EmbeddingSpace,y2(attributeofdog),x1,x2,x3,y1(attributeofchimp),y3(attributeofGrass-mudhorse),1,2,3,2,3,1,Whatifwedonthavedatabase,V(chimp),V(dog),V(Grass-mud_horse),Zero-shotLearning,=min,2,Problem?,Zeroloss:,+max,andasclose,andnotasclose,Zero-shotLearning,ConvexCombinationofSemanticEmbedding,NN,lion,tiger,0.5,0.5,V(lion),V(tiger),V(liger),0.5V(tiger)+0.5V(lion),Findtheclosestwordvector,Onlyneedoff-the-shelfNNforImageNetandwordvector,/pdf/1312.5650v3.pdf,ExampleofZero-shotLearning,MelvinJohnson,MikeSchuster,QuocV.Le,MaximKrikun,YonghuiWu,ZhifengChen,NikhilThorat.GooglesMultilingualNeuralMachineTranslationSystem:EnablingZero-ShotTranslation,arXivpreprint2016,ExampleofZero-shotLearning,MoreaboutZero-shotlearning,MarkPalatucci,DeanPomerleau,GeoffreyE.Hinton,TomM.Mitchell,“Zero-shotLearningwithSemanticOutputCodes”,NIPS2009ZeynepAkata,FlorentPerronnin,ZaidHarchaouiandCordeliaSchmid,“Label-EmbeddingforAttribute-BasedClassification”,CVPR2013AndreaFrome,GregS.Corrado,JonShlens,SamyBengio,JeffDean,MarcAurelioRanzato,TomasMikolov,“DeViSE:ADeepVisual-SemanticEmbeddingModel”,NIPS2013MohammadNorouzi,TomasMikolov,SamyBengio,YoramSinger,JonathonShlens,AndreaFrome,