GradientDescent梯度下降法课件

1、GradientDirectional DerivativesGradient descent (GD): AKA steepest descent (SD)Goal: Minimize a function iteratively based on gradientFormula for GD:Normalized versionWith momentumGradient Descent (GD)Step size or learning rateQuiz!Vanilla GDorExample of Single-Input FunctionsIf n=1, GD reduces to t

2、he problem of going left or right.ExampleAnimation:/?p=gradient.descentEach point/region with zero gradient has a basin of attractionBasin of Attraction in 1DExample of Two-input Functions“Peaks” Functions (1/2)If n=2, GD needs to find a direction in 2D plane.Example: “Peaks” function in MATLABAnima

3、tion: gradientDescentDemo.mGradients is perpendicularto contours, why?3 local maxima3 local minima“Peaks” Functions (2/2)Gradient of the “peaks” functiondz/dx = -6*(1-x)*exp(-x2-(y+1)2) - 6*(1-x)2*x*exp(-x2-(y+1)2) - 10*(1/5-3*x2)*exp(-x2-y2) + 20*(1/5*x-x3-y5)*x*exp(-x2-y2) - 1/3*(-2*x-2)*exp(-(x+1

4、)2-y2)dz/dy = 3*(1-x)2*(-2*y-2)*exp(-x2-(y+1)2) + 50*y4*exp(-x2-y2) + 20*(1/5*x-x3-y5)*y*exp(-x2-y2) + 2/3*y*exp(-(x+1)2-y2)d(dz/dx)/dx = 36*x*exp(-x2-(y+1)2) - 18*x2*exp(-x2-(y+1)2) - 24*x3*exp(-x2-(y+1)2) + 12*x4*exp(-x2-(y+1)2) + 72*x*exp(-x2-y2) - 148*x3*exp(-x2-y2) - 20*y5*exp(-x2-y2) + 40*x5*e

5、xp(-x2-y2) + 40*x2*exp(-x2-y2)*y5 -2/3*exp(-(x+1)2-y2) - 4/3*exp(-(x+1)2-y2)*x2 -8/3*exp(-(x+1)2-y2)*x Each point/region with zero gradient has a basin of attractionBasin of Attraction in 2DRosenbrock FunctionRosenbrock functionMore about this functionAnimation: /?p=gradient.descentDocument on how t

6、o optimize this functionJustification for using momentum termsProperties of Gradient DescentPropertiesNo guarantee for reaching global optimumFeasible for differentiable objective functions (which can have finite number of non-differential points)Performance heavily dependent on starting point and s

7、tep size VariantsUse adaptive step sizesNormalize the gradient by its lengthUse the momentum term to reduce zig-zag pathsUse line minimization at each iterationQuiz!Comparisons of Gradient-based OptimizationGradient descent (GD)Treat all parameters as nonlinearHybrid learning of GD+LSEDistinguish be

8、tween linear and nonlinearConjugate gradient descentTry to reach the minimizing point by assume the objective function is quadraticGauss-Newton (GN) methodLinearize the objective function to treat all parameters as linearLevenberg-Marquardt (LM) methodSwitch smoothly between SD and GNSynonymsLineari

9、zation methodExtended Kalman filter methodConcept:General nonlinear model: y = f(x, q)linearization at q = qnow: y = f(x, qnow)+a1(q1 - q1,now)+a2(q2 - q2,now) + .LSE solution: qnext = qnow + h(ATA)-1ATBGauss-Newton MethodFormulaqnext = qnow + h(ATA+lI)-1ATBEffects of ll small Gauss-Newton methodl big Gradient descentHow to update lGreedy policy Make l smallCautious policy Make l bigLevenberg-Marquardt MethodCan we use GD to find the minimum of f(x)=|x|?What is the gradient of the sigmoid function? Can you express the gradient using the original funct