台大-李宏毅-B站机器学习视频-课件神经网络与深度学习Gradient Descent

GradientDescent,Review:GradientDescent,Instep3,wehavetosolvethefollowingoptimizationproblem:,=argmin,L:lossfunction,:parameters,Supposethathastwovariables1,2,Randomlystartat0=1020,1121=1020101202,1=00,1222=1121111212,2=11,Review:GradientDescent,Startatposition0,Computegradientat0,Moveto1=0-0,Computegradientat1,Moveto2=11,Movement,Gradient,0,1,2,3,0,1,2,3,1,2,Gradient:Loss的等高線的法線方向,GradientDescent,Tip1:Tuningyourlearningrates,LearningRate,No.ofparametersupdates,Loss,Loss,VeryLarge,Large,small,Justmake,Setthelearningratecarefully,Iftherearemorethanthreeparameters,youcannotvisualizethis.,Butyoucanalwaysvisualizethis.,AdaptiveLearningRates,Popular&SimpleIdea:Reducethelearningratebysomefactoreveryfewepochs.Atthebeginning,wearefarfromthedestination,soweuselargerlearningrateAfterseveralepochs,weareclosetothedestination,sowereducethelearningrateE.g.1/tdecay:=+1Learningratecannotbeone-size-fits-allGivingdifferentparametersdifferentlearningrates,Adagrad,Dividethelearningrateofeachparameterbytherootmeansquareofitspreviousderivatives,:rootmeansquareofthepreviousderivativesofparameterw,wisoneparameters,=,VanillaGradientdescent,Adagrad,+1,=+1,+1,Parameterdependent,Adagrad,10000,21111,+1,0=02,1=1202+12,=1+1=02,32222,2=1302+12+22,:rootmeansquareofthepreviousderivativesofparameterw,Adagrad,Dividethelearningrateofeachparameterbytherootmeansquareofitspreviousderivatives,=+1,+1=02,1/tdecay,+1,=1+1=02,Contradiction?,+1=02,VanillaGradientdescent,Adagrad,Largergradient,largerstep,Largergradient,smallerstep,Largergradient,largerstep,+1,=,=+1,IntuitiveReason,Howsurpriseitis,+1=02,造成反差的效果,反差,=,=+1,特別大,特別小,Largergradient,largersteps?,=2+,=|2+|,0,|0+2|,0,|20+|,Beststep:,2,|20+|2,Larger1storderderivativemeansfarfromtheminima,Comparisonbetweendifferentparameters,1,2,a,b,c,d,cd,ab,Larger1storderderivativemeansfarfromtheminima,Donotcrossparameters,SecondDerivative,=2+,=|2+|,2,0,|0+2|,0,|20+|,Beststep:,22=2,|20+|2,Comparisonbetweendifferentparameters,1,2,a,b,c,d,cd,ab,Larger1storderderivativemeansfarfromtheminima,Donotcrossparameters,SmallerSecond,LargerSecond,Largersecondderivative,smallersecondderivative,Usefirstderivativetoestimatesecondderivative,firstderivative2,1,2,largersecondderivative,smallersecondderivative,+1=02,?,GradientDescent,Tip2:StochasticGradientDescent,Makethetrainingfaster,StochasticGradientDescent,GradientDescent,StochasticGradientDescent,Pickanexamplexn,Faster!,=+2,Lossisthesummationoveralltrainingexamples,=+2,Lossforonlyoneexample,Demo,StochasticGradientDescent,GradientDescent,StochasticGradientDescent,Seeallexamples,Seeallexamples,Seeonlyoneexample,Updateafterseeingallexamples,Ifthereare20examples,20timesfaster.,Updateforeachexample,GradientDescent,Tip3:FeatureScaling,FeatureScaling,Makedifferentfeatureshavethesamescaling,Sourceoffigure:http:/cs231n.github.io/neural-networks-2/,=+11+22,1,1,2,2,FeatureScaling,1,2,100,200,LossL,1,2,LossL,1,2,=+11+22,FeatureScaling,1,2,3,mean:,standarddeviation:,Themeansofalldimensionsare0,andthevariancesareall1,Foreachdimensioni:,11,21,12,22,GradientDescent,Theory,Question,Whensolving:Eachtimeweupdatetheparameters,weobtainthatmakessmaller.,=argmax,bygradientdescent,012,Isthisstatementcorrect?,WarningofMath,FormalDerivation,Supposethathastwovariables1,2,How?,L(),Givenapoint,wecaneasilyfindthepointwiththesmallestvaluenearby.,TaylorSeries,Taylorseries:Leth(x)beanyfunctioninfinitelydifferentiablearoundx=x0.,Whenxisclosetox0,sin(x)=,E.g.Taylorseriesforh(x)=sin(x)aroundx0=/4,Theapproximationisgoodaround/4.,MultivariableTaylorSeries,Whenxandyisclosetox0andy0,+somethingrelatedto(x-x0)2and(y-y0)2+,BacktoFormalDerivation,BasedonTaylorSeries:,Iftheredcircleissmallenough,intheredcircle,L(),BacktoFormalDerivation,BasedonTaylorSeries:,Iftheredcircleissmallenough,intheredcircle,L(),Find1and2intheredcircleminimizingL(),constant,d,Simple,right?,Gradientdescenttwovariables,RedCircle:,(Iftheradiusissmall),TominimizeL(),Find1and2intheredcircleminimizingL(),BacktoFormalDerivation,Find1and2yieldingthesmallestvalueofinthecircle,Thisisgradientdescent.,BasedonTaylorSeries:,Iftheredcircleissmallenough,intheredcircle,constant,Notsatisfiediftheredcircle(learni