台大-李宏毅-B站机器学习视频-课件神经网络与深度学习semi v3

Semi-supervisedLearning,Introduction,Supervisedlearning:,=1E.g.:image,:classlabelsSemi-supervisedlearning:,=1,=+Asetofunlabeleddata,usuallyURTransductivelearning:unlabeleddataisthetestingdataInductivelearning:unlabeleddataisnotthetestingdataWhysemi-supervisedlearning?Collectingdataiseasy,butcollecting“labelled”dataisexpensiveWedosemi-supervisedlearninginourlives,Whysemi-supervisedlearninghelps?,Labelleddata,Unlabeleddata,cat,dog,(Imageofcatsanddogswithoutlabeling),Whysemi-supervisedlearninghelps?,Thedistributionoftheunlabeleddatatellussomething.,Usuallywithsomeassumptions,Whoknows?,Outline,Semi-supervisedLearningforGenerativeModel,SupervisedGenerativeModel,Givenlabelledtrainingexamples1,2lookingformostlikelypriorprobabilityP(Ci)andclass-dependentprobabilityP(x|Ci)P(x|Ci)isaGaussianparameterizedbyand,1,2,1|=|11|11+|22,With1,2,1,2,DecisionBoundary,Semi-supervisedGenerativeModel,Givenlabelledtrainingexamples1,2lookingformostlikelypriorprobabilityP(Ci)andclass-dependentprobabilityP(x|Ci)P(x|Ci)isaGaussianparameterizedbyand,DecisionBoundary,Theunlabeleddatahelpre-estimate1,2,1,2,1,2,Semi-supervisedGenerativeModel,Initialization:=1,2,1,2,Step1:computetheposteriorprobabilityofunlabeleddataStep2:updatemodel,1|,1=1+1|,1=111+11|1|,Backtostep1,Dependingonmodel,:totalnumberofexamples1:numberofexamplesbelongingtoC1,Thealgorithmconvergeseventually,buttheinitializationinfluencestheresults.,E,M,Why?,MaximumlikelihoodwithlabelleddataMaximumlikelihoodwithlabelled+unlabeleddata,=,=,+,=|11+|22,(cancomefromeitherC1andC2),Closed-formsolution,Solvediteratively,=1,2,1,2,Semi-supervisedLearningLow-densitySeparation,非黑即白,“Black-or-white”,Self-training,Given:labelleddataset=,=1,unlabeleddataset=+Repeat:TrainmodelfromlabelleddatasetApplytotheunlabeleddatasetObtain,=+Removeasetofdatafromunlabeleddataset,andaddthemintothelabeleddataset,Independenttothemodel,Howtochoosethedatasetremainsopen,Regression?,Pseudo-label,Youcanalsoprovideaweighttoeachdata.,Self-training,Similartosemi-supervisedlearningforgenerativemodelHardlabelv.s.Softlabel,Consideringusingneuralnetwork,Newtargetforis10,Class1,70%Class130%Class1,(networkparameter)fromlabelleddata,Newtargetforis0.70.3,Doesntwork,Itlookslikeclass1,thenitisclass1.,Hard,Soft,Entropy-basedRegularization,Distribution,Good!,Good!,Bad!,=15,Entropyof:Evaluatehowconcentratethedistributionis,=0,=0,=5,=15,Assmallaspossible,=,+,labelleddata,unlabeleddata,Outlook:Semi-supervisedSVM,Findaboundarythatcanprovidethelargestmarginandleasterror,Enumerateallpossiblelabelsfortheunlabeleddata,Semi-supervisedLearningSmoothnessAssumption,近朱者赤，近墨者黑,“Youareknownbythecompanyyoukeep”,SmoothnessAssumption,Assumption:“similar”hasthesameMoreprecisely:xisnotuniform.If1and2arecloseinahighdensityregion,1and2arethesame.,connectedbyahighdensitypath,Sourceofimage:/files/pinwheel.png,1,2,3,1and2havethesamelabel,2and3havedifferentlabels,SmoothnessAssumption,“indirectly”similarwithsteppingstones,(TheexampleisfromthetutorialslidesofXiaojinZhu.),similar?,Notsimilar?,Sourceofimage:,SmoothnessAssumption,Carticles,(TheexampleisfromthetutorialslidesofXiaojinZhu.),SmoothnessAssumption,Carticles,(TheexampleisfromthetutorialslidesofXiaojinZhu.),ClusterandthenLabel,Cluster1,Cluster2,Cluster3,Class1,Class2,Class2,Usingallthedatatolearnaclassifierasusual,Graph-basedApproach,Howtoknow1and2arecloseinahighdensityregion(connectedbyahighdensitypath),Representedthedatapointsasagraph,E.g.Hyperlinkofwebpages,citationofpapers,Graphrepresentationisnaturesometimes.,Sometimesyouhavetoconstructthegraphyourself.,Graph-basedApproach-GraphConstruction,Definethesimilarity,betweenandAddedge:KNearestNeighbore-NeighborhoodEdgeweightisproportionaltos,=2,GaussianRadialBasisFunction:,TheimageisfromthetutorialslidesofAmarnagSubramanyaandParthaPratimTalukdar,Graph-basedApproach,Class1,Class1,Propagatethroughthegraph,Thelabelleddatainfluencetheirneighbors.,x,Graph-basedApproach,Definethesmoothnessofthelabelsonthegraph,=12,2,Smallermeanssmoother,x1,x2,x3,x4,2,3,1,1,1=0,2=1,3=1,4=0,=0.5,=3,Foralldata(nomatterlabelledornot),Graph-basedApproach,Definethesmoothnessofthelabelsonthegraph,=,L:(R+U)x(R+U)matrix,GraphLaplacian,=,y:(R+U)-dimvector,=,=0220301031000110,D=5003000000005001,=12,2,Graph-basedApproach,Definethesmoothnessofthelabelsonthegraph,=,=12,2,Dependingonnetworkparameters,=,+,J.Weston,F.Ratle,andR.Collobert,“Deeplearningviasemi-supervisedembedding,”ICML,2008,Asaregularizationterm,smooth,smooth,smooth,Semi-supervisedLearningBetterRepresentation,去蕪存菁，化繁為簡,LookingforBetterRepresentation,Findthelate