机器学习实验4

1、 实验题目：Regularization实验学时：实验日期： 实验目的：掌握线性回归和逻辑回归的正规化，理解 对参量的范数的影响 软件环境：Octave-4.2.1 (GUI) 实验步骤与内容：要求：第一部分：线性回归的正则化1. 下载数据包 "ex5Linx.dat" 和 "ex5Liny.dat" 。2.本实验中的x是个标量，只有一个特征，用5阶多项式拟合的预测函数：为解决过拟合，引入正规化因子，此时代价函数：3.使用梯度下降法：各自正规化因子对应的预测曲线如下：4.使用正规方程解法：=(XTX)1XTy即：各自正规化因子对应的预测曲线如下

2、：第二部分：逻辑回归的正规化1.下载'ex5Logx.dat' 和 ex5Logy.dat'2.代价方程：其中：采用牛顿法求解最小代价函数。迭代函数：其中：3. 各自正规化因子对应的预测曲线如下：实验结果1. 2. 结论分析与体会： 1.对于线性回归，可以看出，随着 的增大，参量的范数下降。这是由于大的 补偿了原代价函数中大的参数。当 过大时，容易出现欠拟合，且预测曲线的走向与实际的相反。 2.对于逻辑回归，当 增大时， 参量的范数减小。但是大到一定程度后也存在边界欠拟合的状况附录：程序源代码1x = load('ex4Linx.dat');y = lo

3、ad('ex4Liny.dat');x_test = -1: 0.01 : 1'x1 = ones(size(x(:, 1), 1), x, x.2, x.3, x.4, x.5; % m * 6x_test1 = ones(size(x_test(:, 1), 1), x_test, x_test.2, x_test.3, x_test.4, x_test.5;% testm, n = size(x1);theta = zeros(n, 1);iter = 2000;alpha = 0.07;lamda = 0, 1, 10; % regularized paramJ

4、_value = zeros(iter, 1); % cost valueE = eye(n, n);E(1, 1) = 0;norm_gradient = zeros(length(lamda), 1);for lamdaTemp = 1 : length(lamda) theta = zeros(n, 1); for iterTemp = 1 : iter h_theta = x1 * theta; % m * 1 J_value(iterTemp) = 1 / 2 / m * (sum(h_theta - y).2). + lamda(lamdaTemp) .* (sum(theta.2

5、) - theta(1).2); theta = theta - alpha ./ m .* (x1' * (h_theta - y) + lamda(lamdaTemp) * E * theta); %iteration function end figure; scatter(x, y, 'o','LineWidth', 2, 'MarkerEdgeColor','k','MarkerFaceColor','r'); hold on; plot(x_test, x_test1 * the

6、ta, '-b','LineWidth',2); legend('Training data','5th order fit, =' num2str(lamda(lamdaTemp); figure; plot(1: iter, J_value); xlabel('iteration'); ylabel('J_value'); theta norm_gradient(lamdaTemp) = norm(theta);endnorm_gradient% Normal equationsnorm_nor

7、mal = zeros(length(lamda), 1);for lamdaTemp = 1 : length(lamda) theta = pinv(x1' * x1 + lamda(lamdaTemp) .* E) * x1' * y norm_normal(lamdaTemp) = norm(theta); figure; scatter(x, y, 'o','LineWidth', 2, 'MarkerEdgeColor','k','MarkerFaceColor','r'

8、); hold on; plot(x_test, x_test1 * theta, '-b','LineWidth',2); legend('Training data','5th order fit, =' num2str(lamda(lamdaTemp);endnorm_normal2.x = load('ex4Logx.dat');y = load('ex4Logy.dat');% Find the indices for the 2 classespos = find(y = 1); neg

9、 = find(y = 0);g = inline('1.0 ./ (1.0 + exp(-z)'); % Usage: To find the value of the sigmoid degree = 6;lamda = 0, 1, 10;x1 = map_feature(x(:,1), x(:,2), degree); % m * nm, n = size(x1);E = ones(n, 1);E(1) = 0;norm_lamda = zeros(length(lamda),1);for lamdaTemp = 1 : length(lamda) theta = zer

10、os(n, 1); J_theta = 0; thetaTemp = zeros(n, 1); J_thetaTemp = 0; while (1) h_theta = g(x1 * thetaTemp); % m * 1 J_thetaTemp = -1 ./ m * (sum(y .* log(h_theta) + (1 - y) .* log(1 - h_theta). - lamda(lamdaTemp) ./ 2 * sum(thetaTemp.2) - thetaTemp(1).2) if (abs(J_theta - J_thetaTemp) < 0.0001) theta

11、 = thetaTemp break; end J_theta = J_thetaTemp; H = 1 ./ m * (x1' * diag(h_theta .*(1 - h_theta) * x1 + lamda(lamdaTemp) .* diag(E); % n * n delta_J = 1 ./ m * (x1' * (h_theta - y) + lamda(lamdaTemp) .* diag(E) * thetaTemp); % n * 1 thetaTemp = thetaTemp - pinv(H) * delta_J; end norm_lamda(la

12、mdaTemp) = norm(theta); figure; plot(x(pos, 1), x(pos, 2), '+', 'MarkerEdgeColor','k','MarkerFaceColor','k','MarkerSize',6); hold on; plot(x(neg, 1), x(neg, 2), 'o', 'MarkerEdgeColor','k','MarkerFaceColor','r',&#

13、39;MarkerSize',6); % % Define the ranges of the grid u = linspace(-1, 1.5, 200); v = linspace(-1, 1.5, 200); % Initialize space for the values to be plotted z = zeros(length(u), length(v); % Evaluate z = theta*x over the grid for i = 1:length(u) for j = 1:length(v) % Notice the order of j, i here! z(j,i) = map_feature(u(i), v(j)*theta; end end % Because of the way that contour plotting works % in Matlab, we need to transpose z, or % else the axis orientation will be flipped! %z = z' % Plot z = 0 by specifying the range 0, 0 hold on; co