机器学习实验4.doc

.实验题目：Regularization实验学时：实验日期： 实验目的：掌握线性回归和逻辑回归的正规化，理解 对参量的范数的影响 软件环境：Octave-4.2.1 (GUI) 实验步骤与内容：要求：第一部分：线性回归的正则化1. 下载数据包 ex5Linx.dat 和 ex5Liny.dat 。2.本实验中的x是个标量，只有一个特征，用5阶多项式拟合的预测函数：为解决过拟合，引入正规化因子，此时代价函数：3.使用梯度下降法：各自正规化因子对应的预测曲线如下：4.使用正规方程解法：=(XTX)1XTy即：各自正规化因子对应的预测曲线如下：第二部分：逻辑回归的正规化1.下载ex5Logx.dat 和 ex5Logy.dat2.代价方程：其中：采用牛顿法求解最小代价函数。迭代函数：其中：3. 各自正规化因子对应的预测曲线如下：实验结果1. 2. 结论分析与体会： 1.对于线性回归，可以看出，随着 的增大，参量的范数下降。这是由于大的 补偿了原代价函数中大的参数。当 过大时，容易出现欠拟合，且预测曲线的走向与实际的相反。 2.对于逻辑回归，当 增大时， 参量的范数减小。但是大到一定程度后也存在边界欠拟合的状况附录：程序源代码1x = load(ex4Linx.dat);y = load(ex4Liny.dat);x_test = -1: 0.01 : 1;x1 = ones(size(x(:, 1), 1), x, x.2, x.3, x.4, x.5; % m * 6x_test1 = ones(size(x_test(:, 1), 1), x_test, x_test.2, x_test.3, x_test.4, x_test.5;% testm, n = size(x1);theta = zeros(n, 1);iter = 2000;alpha = 0.07;lamda = 0, 1, 10; % regularized paramJ_value = zeros(iter, 1); % cost valueE = eye(n, n);E(1, 1) = 0;norm_gradient = zeros(length(lamda), 1);for lamdaTemp = 1 : length(lamda) theta = zeros(n, 1); for iterTemp = 1 : iter h_theta = x1 * theta; % m * 1 J_value(iterTemp) = 1 / 2 / m * (sum(h_theta - y).2). + lamda(lamdaTemp) .* (sum(theta.2) - theta(1).2); theta = theta - alpha ./ m .* (x1 * (h_theta - y) + lamda(lamdaTemp) * E * theta); %iteration function end figure; scatter(x, y, o,LineWidth, 2, MarkerEdgeColor,k,MarkerFaceColor,r); hold on; plot(x_test, x_test1 * theta, -b,LineWidth,2); legend(Training data,5th order fit, = num2str(lamda(lamdaTemp); figure; plot(1: iter, J_value); xlabel(iteration); ylabel(J_value); theta norm_gradient(lamdaTemp) = norm(theta);endnorm_gradient% Normal equationsnorm_normal = zeros(length(lamda), 1);for lamdaTemp = 1 : length(lamda) theta = pinv(x1 * x1 + lamda(lamdaTemp) .* E) * x1 * y norm_normal(lamdaTemp) = norm(theta); figure; scatter(x, y, o,LineWidth, 2, MarkerEdgeColor,k,MarkerFaceColor,r); hold on; plot(x_test, x_test1 * theta, -b,LineWidth,2); legend(Training data,5th order fit, = num2str(lamda(lamdaTemp);endnorm_normal2.x = load(ex4Logx.dat);y = load(ex4Logy.dat);% Find the indices for the 2 classespos = find(y = 1); neg = find(y = 0);g = inline(1.0 ./ (1.0 + exp(-z); % Usage: To find the value of the sigmoid degree = 6;lamda = 0, 1, 10;x1 = map_feature(x(:,1), x(:,2), degree); % m * nm, n = size(x1);E = ones(n, 1);E(1) = 0;norm_lamda = zeros(length(lamda),1);for lamdaTemp = 1 : length(lamda) theta = zeros(n, 1); J_theta = 0; thetaTemp = zeros(n, 1); J_thetaTemp = 0; while (1) h_theta = g(x1 * thetaTemp); % m * 1 J_thetaTemp = -1 ./ m * (sum(y .* log(h_theta) + (1 - y) .* log(1 - h_theta). - lamda(lamdaTemp) ./ 2 * sum(thetaTemp.2) - thetaTemp(1).2) if (abs(J_theta - J_thetaTemp) 0.0001) theta = thetaTemp break; end J_theta = J_thetaTemp; H = 1 ./ m * (x1 * diag(h_theta .*(1 - h_theta) * x1 + lamda(lamdaTemp) .* diag(E); % n * n delta_J = 1 ./ m * (x1 * (h_theta - y) + lamda(lamdaTemp) .* diag(E) * thetaTemp); % n * 1 thetaTemp = thetaTemp - pinv(H) * delta_J; end norm_lamda(lamdaTemp) = norm(theta); figure; plot(x(pos, 1), x(pos, 2), +, MarkerEdgeColor,k,MarkerFaceColor,k,MarkerSize,6); hold on; plot(x(neg, 1), x(neg, 2), o, MarkerEdgeColor,k,MarkerFaceColor,r,MarkerSize,6); % % Define the ranges of the grid u = linspace(-1, 1.5, 200); v = linspace(-1, 1.5, 200); % Initialize space for the values to be plotted z = zeros(length(u), length(v); % Evaluate z = theta*x over the grid for i = 1:length(u) for j = 1:length(v) % Notice the order of j, i here! z(j,i) = map_feature(u(i), v(j)*theta; end end % Because of the way that contour plotting works % in Matlab, we need to transpose z, or % else the axis orientation will be flipped! %z = z; % Plot z = 0 by sp