面向缺陷的软件系统可靠管理规范的研究.ppt

1、a,1,面向缺陷的软件系统可靠性管理规范的研究,寇纲 电子科技大学经济与管理学院,a,2,a,3,基于数据挖掘和多目标决策的软件风险评估和管理,Risk Assessment,Risk Management,What can be done and what options are available? What are the associated trade-offs in terms of all costs, benefits, and risks? What are the impacts of current management decisions on future optio

2、ns,What can go wrong? What is the likelihood that it could go wrong? What are the consequences? What is the time domain,Kaplan and Garrick 1981,Haimes 1991,Risk Communication (Data Mining,Risk Communication (MCDM,Yacov Haimes 2009,a,4,No Free Lunch (NFL) theorem,if algorithm A outperforms algorithm

3、B on some cost functions, then loosely speaking there must exist exactly as many other functions where B outperforms A” (Wolpert and Macready, 1995). In other words, there exists no single classifier that could achieve the best performance for all measures,a,5,Approach 1overview,Aim: Design a perfor

4、mance metric that combines various measures to evaluate the quality of classifiers for software defect prediction; Data: 11 datasets from NASA MDP repository; Tool: WEKA Techniques: Statistic,a,6,Approach 1Classifiers,Trees: classification and regression tree (CART), Nave Bayes tree, and C4.5 Functi

5、ons: linear logistic regression, radial basis function (RBF) network, sequential minimal optimization (SMO), Support Vector Machine (SVM), and Neural Networks Bayesian classifiers: Bayesian network and Nave Bayes lazy classifiers: K-nearest-neighbor Rules: decision table and Repeated Incremental Pru

6、ning to Produce Error Reduction (RIPPER) rule induction,a,7,Approach 1Step 1,For a specific dataset i (i=1,2,11), and a specific performance measure j (j=1,2,13), Do t test for pairs of classifiers (k=1, 2,13): (the statistical significance is set as 0.05,I f C_1 performs better at measure j than C_

7、2,The top three ranking classifiers are assigned to the score of 3, 2, and 1, respectively,a,8,Approach 1Step 2,For a specific dataset i,The larger the “Sum_rank”, the better the classifier is. The value of “Sum_rank” is normalized,Sum,a,9,Approach 1Step 3,For a specific dataset i,Sum,The lager the

8、score, the better the classifier,a,10,Approach 1Results,a,11,Approach 1conclusion,The best result for a given dataset according to a given measure may perform poorly on a different measure. Neural network and SVM have longer training time than other classifiers in general. No classifier yielding the

9、 best measures across the 11 datasets. SVM (functions.LibSVM), K-nearest-neighbor (lazy.IBk), and C4.5 (trees.J48) ranked the top three classifiers based on the experiment,a,12,Approach 2why,Experimental results have shown that ensemble of classifiers are often more accurate and robust to the effect

10、s of noisy data, and achieve lower average error rate than any of the constituent classifiers. However, inconsistencies exist in different studies and the performances of learning algorithms may vary using different performance measures and under different circumstances,a,13,Approach 2Overview,Aim:

11、Evaluate the performance of ensemble classifiers for software defect detection; Data: 11 datasets from NASA MDP repository; Tool: WEKA, Matlab 7.0 Techniques: MCDM Tool: AHP,a,14,Approach 2ensemble methods,Bagging It combines multiple algorithms by taking a plurality vote to get an aggregated single

12、 predictor. randomly sampling. Boosting In boosting, however, weights of training instances change in each iteration to force learning algorithms to put more emphasis on instances that were predicted incorrectly previously and less emphasis on instances that were predicted correctly previously Stack

13、ing minimizing the generalization error rate of one or more algorithms; Can different types of learning algorithms; Vote,a,15,Approach 2AHP,The analytic hierarchy process (AHP) is a multi-criteria decision making approach that helps decision makers structure a decision problem based on pairwise comp

14、arisons and experts judgments,a,16,Approach 2Pairwise comparisons of performance measures,a,17,Approach 2Priorities of AdaBoost classifiers (Group 1,a,18,Approach 2 Priorities of bagging classifiers (Group 2,a,19,Approach 2 Priorities of stacking, voting and individual classifiers (Group 3,a,20,Appr

15、oach 2 Priorities of classifiers of the top 5 classifiers from each group,a,21,Approach 2 Conclusions,Ensemble methods can improve the classification results for software defect prediction in general; AdaBoost ensemble method gives the best results. Tree and rule based classifiers perform better tha

16、n other types of classifiers in the experiment. Stacking and voting can improve classification results and provide relatively stable outcomes, but the results are not as good as AdaBoost and bagging. The ranking conducted by the AHP may change in different settings of comparisons. When the set of al

17、ternative classifiers change, the relative ranking of algorithms may change, especially when the difference between two classifiers is statistically significant,a,22,Approach 3why,The ranking conducted by the AHP may change in different settings of comparisons. There are lots of MCDM methods availab

18、le to evaluate the classification results. We want to find out whether they make the same choice,a,23,Approach 3MCDM Methods,DEA Data envelopment analysis They evaluate the efficiency of decision making units (DMUs) through identifying the efficiency frontier and comparing each DMU with the frontier

19、; CCR PROMETHEE II Preference Ranking Organisation METHod for Enrichment of Evaluations,a,24,Approach 3weights,Weights reflect the preference of decision makers,a,25,Approach 3Results of the DEA technique,a,26,Approach 3Results of the TOPSIS technique,a,27,Approach 3Results of the ELECTRE I techniqu

20、e,a,28,Approach 3Results of the PROMETHEE II technique,a,29,Approach 3 Conclusion,The boosting of CART and the boosting of C4.5 decision tree are ranked as the most appropriate algorithms for software defect datasets. The four MCDM methods generate similar top-ranked classification algorithms while

21、produce different ranking for some classifiers for the selected software defect datasets. TOPSIS and PROMETHEE II may be more appropriate than DEA and ELECTRE I for the given task since they provide a complete ranking of algorithms,a,30,Approach 4 FAMCDM: A Fusion Approach of MCDM Methods to Rank Mu

22、lticlass Classification Algorithms,Since multiclass algorithms selection normally involves more than one criterion, such as accuracy and computation time, the selection process can be modeled as a multiple criteria decision making (MCDM) problem. While the evaluations of algorithms provided by vario

23、us MCDM methods are in agreement most of the time, there are situations where different MCDM methods generate very different results. To resolve this disagreement and help the decision maker to make the choices, this paper proposes a fusion approach to produce a weighted compatible ranking of multic

24、lass classification algorithms. A large-scale multiclass network intrusion prediction task is used as an illustrative case. The results of the experimental study suggest that MCDM methods are useful tools for evaluating multiclass classification algorithms and the fusion approach proposed is capable

25、 to identify a compromised solution in conflicting rankings generated by different MCDM methods,a,31,Approach 4 FAMCDM: A Fusion Approach of MCDM Methods to Rank Multiclass Classification Algorithms,In the first step, a selection of MCDM methods is applied to rank multiclass classification algorithm

26、s. If there are strong disagreements among MCDM methods, the different ranking scores generated by MCDM methods are used as inputs for the second step. The goal of the second step is to determine the weights of different MCDM methods. This paper utilizes the combinational evaluation model to find the weights for each MCDM method. The third step of the fusion approach uses the weights obtained from the second step to get secondary rankings of algorithms. Rankin