交叉验证我的总结.doc

先说下我和邵是云的聊天情况：她的意思其实一开始所有样本也是先分为了两个部分，一个大的部分是训练集，一个小的部分是测试集，然后只是在训练集里面分为常规训练集和一个效验集，且是交叉验证的方式，都全部交叉验证搞完了，再最后单独测试那个小部分的测试集样本？就是你说的training accuracy（常规训练集）, cross validation rate（效验集） and test accuracy（测试集），她说这是标准的方式，而如果全体数据用来训练和交叉验证其实就是把所有的样本全部分为了训练集和效验集，所以就没有test accuracy一说。常用的精度测试方法有交叉验证，例如10倍交叉验证(10-fold cross validation)，将数据集分成十分，轮流将其中9份做训练1份做测试，10次的结果的均值作为对算法精度的估计，一般还需要进行多次10倍交叉验证求均值，例如10次10倍交叉验证，更精确一点。当训练样本数目过少，采用“交叉验证法”（cross validation）.交叉验证法分两种1:K重交叉验证法：该方法是最为普遍的计算推广误差的方法之一。其过程为：将训练样本集随机分为K个集合，通常分为K等份，对其中的K-1个集合进行训练，得到一个决策函数，并用决策函数对剩下的一个集合进行样本测试。该过程重复K次，取K次过程中的测试错误的平均值作为推广误差。2:留一法：该方法可以说是K重交叉验证法的极端情况，即K=L，L为整个训练样本集的大小。该过程为 ：对于第i个训练样本，将其取出，对剩下L-1个样本进行训练，得到决策函数，并用其测试第i个训练样本，该过程重复L次，用此方法求出的误差对于实际中的测试误差来说几乎是无偏的。(注意当样本过少,即使交叉验证效果也不会理想,一般样本应在100以上.)k-foldvalidation中文就是k-折交叉验证(确认)其中的k是用户自己定的但它必须比原始的训练集中的元素个数n要小,即k=1的条件吗），即经验风险Remp 为 0 的前提下，通过对分类间隔最大化（咦，这个就是（w）(1/2)*w*w嘛），使分类器获得最好的推广性能。 那么解释完线性可分了，我们知道其实很多时候是线性不可分的啊，那么有什么区别没有啊？废话区别当然会有啦，嘿嘿那么什么是本质的区别啊？本质的区别就是不知道是否线性可分但是允许有错分的样本存在（这个咋回事还是没明白hoho）但是正是由于允许存在错分样本，此时的软间隔分类超平面表示在剔除那些错分样本后最大分类间隔的超平面。这里就出现了新词松驰因子，干吗用滴？就是用来控制错分样本的啊。这样的话经验风险就要跟松驰因子联系在一起了。而C就是松驰因子前面的系数，C0 是一个自定义的惩罚因子，它控制对错分样本惩罚的程度，用来控制样本偏差与机器推广能力之间的折衷。c越小，惩罚越小，那么训练误差就越大，使得结构风险也变大，而C 越大呢，惩罚就越大，对错分样本的约束程度就越大，但是这样会使得第二项置信范围的权重变大那么分类间隔的权重就相对变小了，系统的泛化能力就变差了。所以选择合适的C还是很有必要的。选择核函数。核函数有很多种，如线性核、多项式核、Sigmoid 核和 RBF（Radial Basis function）核。本文选定 RBF 核为 SVM 的核函数（RBF 核K(x, y) = exp( | x y |的平方), 0）。因为RBF 核可以将样本映射到一个更高维的空间，可以处理当类标签（Class Labels）和特征之间的关系是非线性时的样例。Keerthi 等25证明了一个有惩罚参数C 的线性核同有参数(C, )（其中C 为惩罚因子， 为核参数）的 RBF 核具有相同的性能。对某些参数，Sigmoid核同 RBF 核具有相似的性能26。另外，RBF 核与多项式核相比具有参数少的优点。因为参数的个数直接影响到模型选择的复杂性。非常重要的一点是01)或者0 xi xj + r Open - a1.mblEdit - Metacode - A90:9Model - LOOPredictEdit - Metacode - L90:9Model - LOOPredictEdit - Metacode - L10:9Model - LOOPredictLOOPredict goes through the entire data set and makes LOO predictions for each point. At the bottom of the page it shows the summary statistics including Mean LOO error, RMS LOO error, and information about the data point with the largest error. The mean absolute LOO-XVEs for the three metacodes given above (the same three used to generate the graphs in fig. 25), are 2.98, 1.23, and 1.80. Those values show that global linear regression is the best metacode of those three, which agrees with our intuitive feeling from looking at the plots in fig. 25. If you repeat the above operation on data set b1.mbl youll get the values 4.83, 4.45, and 0.39, which also agrees with our observations. What are cross-validation and bootstrapping? -Cross-validation and bootstrapping are both methods for estimatinggeneralization error based on resampling (Weiss and Kulikowski 1991; Efronand Tibshirani 1993; Hjorth 1994; Plutowski, Sakata, and White 1994; Shaoand Tu 1995). The resulting estimates of generalization error are often usedfor choosing among various models, such as different network architectures. Cross-validation+In k-fold cross-validation, you divide the data into k subsets of(approximately) equal size. You train the net k times, each time leavingout one of the subsets from training, but using only the omitted subset tocompute whatever error criterion interests you. If k equals the samplesize, this is called leave-one-out cross-validation. Leave-v-out is amore elaborate and expensive version of cross-validation that involvesleaving out all possible subsets of v cases. Note that cross-validation is quite different from the split-sample orhold-out method that is commonly used for early stopping in NNs. In thesplit-sample method, only a single subset (the validation set) is used toestimate the generalization error, instead of k different subsets; i.e.,there is no crossing. While various people have suggested thatcross-validation be applied to early stopping, the proper way of doing so isnot obvious. The distinction between cross-validation and split-sample validation isextremely important because cross-validation is markedly superior for smalldata sets; this fact is demonstrated dramatically by Goutte (1997) in areply to Zhu and Rohwer (1996). For an insightful discussion of thelimitations of cross-validatory choice among several learning methods, seeStone (1977). Jackknifing+Leave-one-out cross-validation is also easily confused with jackknifing.Both involve omitting each training case in turn and retraining the networkon the remaining subset. But cross-validation is used to estimategeneralization error, while the jackknife is used to estimate the bias of astatistic. In the jackknife, you compute some statistic of interest in eachsubset of the data. The average of these subset statistics is compared withthe corresponding statistic computed from the entire sample in order toestimate the bias of the latter. You can also get a jackknife estimate ofthe standard error of a statistic. Jackknifing can be used to estimate thebias of the training error and hence to estimate the generalization error,but this process is more complicated than leave-one-out cross-validation(Efron, 1982; Ripley, 1996, p. 73). Choice of cross-validation method+Cross-validation can be used simply to estimate the generalization error ofa given model, or it can be used for model selection by choosing one ofseveral models that has the smallest estimated generalization error. Forexample, you might use cross-validation to choose the number of hiddenunits, or you could use cross-validation to choose a subset of the inputs(subset selection). A subset that con