台大-李宏毅-B站机器学习视频-课件神经网络与深度学习DNN tip

TipsforDeepLearning,NeuralNetwork,GoodResultsonTestingData?,GoodResultsonTrainingData?,YES,YES,NO,NO,Overfitting!,RecipeofDeepLearning,DonotalwaysblameOverfitting,DeepResidualLearningforImageRecognition/abs/1512.03385,TestingData,Overfitting?,TrainingData,Notwelltrained,NeuralNetwork,GoodResultsonTestingData?,GoodResultsonTrainingData?,YES,YES,RecipeofDeepLearning,Differentapproachesfordifferentproblems.,e.g.dropoutforgoodresultsontestingdata,GoodResultsonTestingData?,GoodResultsonTrainingData?,YES,YES,RecipeofDeepLearning,HardtogetthepowerofDeep,Deeperusuallydoesnotimplybetter.,ResultsonTrainingData,VanishingGradientProblem,Largergradients,Almostrandom,Alreadyconverge,basedonrandom!?,Learnveryslow,Learnveryfast,Smallergradients,VanishingGradientProblem,Intuitivewaytocomputethederivatives,=?,+,+,Smallergradients,ReLU,RectifiedLinearUnit(ReLU),Reason:,1.Fasttocompute,2.Biologicalreason,3.Infinitesigmoidwithdifferentbiases,4.Vanishinggradientproblem,XavierGlorot,AISTATS11,AndrewL.Maas,ICML13,KaimingHe,arXiv15,ReLU,0,0,0,0,ReLU,AThinnerlinearnetwork,Donothavesmallergradients,ReLU-variant,alsolearnedbygradientdescent,Maxout,LearnableactivationfunctionIanJ.Goodfellow,ICML13,Max,Input,Max,7,1,Max,Max,2,4,ReLUisaspecialcasesofMaxout,Youcanhavemorethan2elementsinagroup.,neuron,Maxout,0,0,=+,1=+,2=0,ReLUisaspecialcasesofMaxout,Maxout,=+,1=+,2=+,LearnableActivationFunction,MorethanReLU,Maxout,LearnableactivationfunctionIanJ.Goodfellow,ICML13ActivationfunctioninmaxoutnetworkcanbeanypiecewiselinearconvexfunctionHowmanypiecesdependingonhowmanyelementsinagroup,2elementsinagroup,3elementsinagroup,Maxout-Training,Givenatrainingdatax,weknowwhichzwouldbethemax,Max,Input,Max,11,21,Max,Max,12,22,1,2,11,21,Maxout-Training,Givenatrainingdatax,weknowwhichzwouldbethemaxTrainthisthinandlinearnetwork,Input,11,21,12,22,1,2,Differentthinandlinearnetworkfordifferentexamples,GoodResultsonTestingData?,GoodResultsonTrainingData?,YES,YES,RecipeofDeepLearning,Review,LargerLearningRate,SmallerLearningRate,Adagrad,+1=02,Usefirstderivativetoestimatesecondderivative,RMSProp,1,2,ErrorSurfacecanbeverycomplexwhentrainingNN.,LargerLearningRate,SmallerLearningRate,RMSProp,1000,2111,+1,0=0,1=02+112,3222,2=12+122,=12+12,RootMeanSquareofthegradientswithpreviousgradientsbeingdecayed,Hardtofindoptimalnetworkparameters,TotalLoss,Thevalueofanetworkparameterw,Veryslowattheplateau,Stuckatlocalminima,=0,Stuckatsaddlepoint,=0,0,Inphysicalworld,Momentum,Howaboutputthisphenomenoningradientdescent?,Review:VanillaGradientDescent,Startatposition0,Computegradientat0,Moveto1=0-0,Computegradientat1,Moveto2=11,Movement,Gradient,0,1,2,3,0,1,2,3,Stopuntil0,Momentum,Startatpoint0,Computegradientat0,Moveto1=0+v1,Computegradientat1,Movementv0=0,Movementv1=v0-0,Movementv2=v1-1,Moveto2=1+v2,Movement,Gradient,0,1,2,3,0,1,2,3,Movementnotjustbasedongradient,butpreviousmovement.,Movementoflaststep,Movement:movementoflaststepminusgradientatpresent,Momentum,viisactuallytheweightedsumofallthepreviousgradient:0,1,1,v0=0,v1=-0,v2=-0-1,Startatpoint0,Computegradientat0,Moveto1=0+v1,Computegradientat1,Movementv0=0,Movementv1=v0-0,Movementv2=v1-1,Moveto2=1+v2,Movementnotjustbasedongradient,butpreviousmovement,Movement:movementoflaststepminusgradientatpresent,Movement=Negativeof+Momentum,Momentum,cost,=0,Stillnotguaranteereachingglobalminima,butgivesomehope,Adam,RMSProp+Momentum,formomentum,forRMSprop,GoodResultsonTestingData?,GoodResultsonTrainingData?,YES,YES,RecipeofDeepLearning,EarlyStopping,Epochs,TotalLoss,Trainingset,Testingset,Stopathere,Validationset,GoodResultsonTestingData?,GoodResultsonTrainingData?,YES,YES,RecipeofDeepLearning,Regularization,NewlossfunctiontobeminimizedFindasetofweightnotonlyminimizingoriginalcostbutalsoclosetozero,Originalloss(e.g.minimizesquareerror,crossentropy),(usuallynotconsiderbiases),Regularizationterm,L2regularization:,Regularization,Newlossfunctiontobeminimized,Gradient:,Update:,Closertozero,WeightDecay,L2regularization:,Regularization,Newlossfunctiontobeminimized,Update:,Alwaysdelete,L1regularization:,L2,Regularization-WeightDecay,Ourbrainprunesouttheuselesslinkbetweenneurons.,Doingthesamethingtomachinesbrainimprovestheperformance.,GoodResultsonTestingData?,GoodResultsonTrainingData?,YES,YES,RecipeofDeepLearning,Dropout,Training:,Eachtimebeforeupdatingtheparameters,Eachneuronhasp%todropout,Dropout,Training:,Eachtimebeforeupdatingtheparameters,Eachneuronhasp%todropout,Usingthenewnetworkfortraining,Thestructureofthenetworkischanged.,Thinner!,Foreachmini-batch,weresamplethedropoutneurons,Dropout,Testing:,Nodropout,Ifthedropoutrateattrainingisp%,alltheweightstimes1-p%,Assumethatthedropoutrateis50%.Ifaweightw=1bytraining,set=0.5fortesting.,Dropout-IntuitiveReason,Training,Testing,Dropout(腳上綁重物),Nodropout(拿下重物後就變很強),Dropout-IntuitiveReason,Whenteamsup,ifeveryoneexpectthepartnerwilldothework,nothingwillbedonefinally.,However,ifyouknowyourpartnerwilldropout,youwilldobetter.,我的partner會擺爛，所以我要好好做,Whentesting,noonedropoutactually,soobtaininggoodresultseventually.,Dropout-IntuitiveReason,Whytheweightsshouldmultiply(1-p)%(dropoutrate)whentesting?,TrainingofDropout,TestingofDropout,1,2,3,4,1,2,3,4,Assumedropoutrateis50%,0.5,0.5,0.5,0.5,Nodropout,Dropoutisakindofensemble.,Ensemble,Network1,Network2,Network3,Network4,Trainabunchofnetworkswithdifferentstructures,TrainingSet,Set1,Set2,Set3,Set4,Dropoutisakindofensemble.,Ensemble,y1,Network1,Network2,Network3,Network4,Testingdatax,y2,y3,y4,average,Dropoutisakindofensemble.,TrainingofDropout,minibatch1,Usingonemini-batchtotrainonenetwork,Someparametersinthenetworkareshared,minibatch2,minibatch3,minibatch4,Mneurons,2Mpossiblenetwork