lecture10-软件测试相关度量.ppt

1,Software Metrics,Lecture 10 Software Test Metrics,清 华 大 学 软 件 学 院 2004年5月11日,2,COCOMO,where a, b, c and d depend on the development mode.,Organic mode Semi-detached mode Embedded mode,Basic model Intermediate model Advanced model,3,COCOMO: Basic,The Basic COCOMO,development time cannot be predicted at this stage.,4,COCOMO: Intermediate,The Intermediate COCOMO:,5,COCOMO II,Application Composition E = OP / PROD Early Design model E = 2.45 KLOC EAF Post-Architecture model,6,Putnams SLIM,7,Test Concepts Estimating Number of Test Case & Allocating Test Times Test Coverage Metrics Remaining Defects Metrics,Contents,8,Section 1 Test Concepts,9,防不胜防的软件错误 例1：1963年, 美国, 飞往火星的火箭爆炸, 损失$ 10 million. 原因: FORTRAN循环 DO 5 I = 1, 3 误写为 DO 5 I = 1.3,软件测试是保证软件质量的关键步骤，是对软件规格说明、设计和编码的最后复审，其工件量约占总工作量40%以上（对于人命关天的情况，测试相当于其它部分总成本的3 5倍）,Roles of Software Testing,10,What is software testing,“ Testing is the process of executing a program with the intent of finding errors.” Glen Myers,11,Software Testing Objectives,Needing minimum effort and time Showing that software functions work according to specification,It cannot show absence of errors or defects. It can only show that errors are present,Uncovering different cases of errors,12,1:5-10 Mainframes i.e.,1 tester for every 5 to 10 developers 2:3 Microsoft, 1992 2:1 Lotus (for 1-2-3 Windows) 1:2 Average of 4 major companies,1992 Microsoft, Borland, WordPerfect, Novell,Tester-to-developer ratios,13,What is a test case?,A set of test inputs, execution conditions, and expected results developed for a particular objective,14,A test is made up of (many) test cases A test procedure is the detailed instructions for setting up, starting, monitoring and restarting a given test case. a.k.a test plan a test case may be used in more than one test procedure a test case may include many subtests,What is a test?,15,Test,test case,test case,test case,subtest,subtest,subtest,subtest,What is a test? (cont.),16,Testing Methodology,White-box,Black-box,17,Objective: Cover the software structure,Structural (White-Box) Test Criteria,Criteria based on control flow data flow Metric: percentage of coverage achieved Adequacy based on metric requirements for criteria,18,Graph representation of control flow and data flow relationships,Flow Graphs,Control Flow The partial order of statement execution, as defined by the semantics of the language Data Flow The flow of values from definitions of a variable to its uses,19,Statement Coverage Branch Coverage Condition Coverage Basis Path Coverage All-Edges Coverage All-Paths Coverage,White-Box Test,20,1 2 3 4 5 6 7 8 9 10 11 12 13 14 15,function P return INTEGER is begin X, Y: INTEGER; READ(X); READ(Y); while (X 10) loop X := X 10; exit when X = 10; end loop; if (Y 20 and then X mod 2 = 0) then Y := Y + 20; else Y := Y 20; end if; return 2*X + Y; end P;,A Sample Program to Test,21,2,3,4,5,6,9,10,12,14,T,T,F,F,9,T,F,7,T,F,9a,9b,Ps Control Flow Graph (CFG),22,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,At least 2 test cases needed,(X = 20, Y = 10) (X = 20, Y = 30) ,All-Statements Coverage of P,23,minimum level coverage using structural testing may be impossible to achieve 100% statement coverage code only executed in exceptional or dangerous circumstances code that is unreachable,Statement Testing: weaknesses,24,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,At least 2 test cases needed,(X = 20, Y = 10) (X = 15, Y = 30) ,All-Branches Coverage of P,25,Branch Testing：weaknesses,Branch testing is one level up from statement testing in degree of coverage same problem in achieving 100% coverage,26,2,3,4,5,6,9b,10,12,14,T,T,F,F,9a,T,F,7,T,F,At least 3 test cases needed,(X = 20, Y = 20) 5T,7T,9aF,9bT (X = 15, Y = 10) 5T,7F,9aT,9bF (X = 9, Y = 10) 5F,7F,9aT,9bF,Condition Coverage of P,X 10 X = 10 Y 20 X mod 2=0,27,Condition Coverage,Generate test data such that all conditions in a decision take on both outcomes (if possible) at least once May not achieve branch coverage: consider while (x 20 ) x = 0 causes first true, second false x = 34 causes first false, second true,28,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,Basic-paths Coverage of P,cyclomatic complexity: V(G) = e - n + 2,29,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,4 test cases needed,(X = 10, Y = 10) (X = 10, Y = 20) (X = 20, Y = 10) (X = 15, Y = 10),Basic-paths Coverage of P,30,Basis path testing：Strengths/weaknesses,Basis set is easy to compute the technique is readily applicable to both structured and unstructured programs Drawback: the basis set is not unique; there will be obvious paths omitted Drawback: the number of paths in the basis set can greatly exceed the number necessary to achieve branch coverage,31,2,3,4,5,6,9b,10,12,14,T,T,F,F,9a,T,F,7,T,F,At least 3 test cases needed,(X = 20, Y = 10) (X = 5, Y = 30) (X = 21, Y = 10) ,All-Edges Coverage of P,32,2,3,4,5,6,9b,10,12,14,T,T,F,F,9a,T,F,7,T,F,Infinitely many test cases needed,(X = 5, Y = 10) (X = 15, Y = 10) (X = 25, Y = 10) ,All-Paths Coverage of P,33,All-Paths testing,Generate test data to exercise all distinct paths in a program! a path that makes i iterations through a loop is distinct from a path that makes i+1 iterations through the loop, even if the same nodes are visited in both iterations. Thus, there are an infinite number of paths in the string processing program!,34,All-Paths testing (cont.),包含的不同执行路径数达520条，对每一条路径进行测试需要1毫秒，假定一年工作365 24小时，要想把所有路径测试完，需3170年,35,2,3,4,5,6,9b,10,12,14,T,F,9a,T,F,Y,X,X,Y,Y,X,X,Y,Y,X,X,X,T,F,7,T,F,X,X,Ps Control and Data Flow Graph,36,How many test cases are needed?,2,3,4,5,6,9b,10,12,14,T,F,9a,T,F,Y,X,X,Y,Y,X,X,Y,Y,X,X,X,T,F,7,T,F,X,X,All-Uses Coverage of P,37,all-statements,all-edges,boundary-interior loop testing,min-max loop testing,all-paths,all-defs,all-uses,all-DU-paths,all-p-uses,all-c-uses,C1,C2,subsumes,Structural Subsumption Hierarchy,all-branches,38,Black Box Test,Equivalence Class Boundary Analysis ,39,Testing Levels,Testing occurs throughout lifecycle Unit Integration System Acceptance Regression,40,Overall Approach,Unit testing (module) Integration testing System testing Acceptance testing,White / Black box Black box Black box - requirements Black box - performance,41,Section 2 Estimating Number of Test Case & Allocating Test Times,42,Estimate No. of Test Cases,Time (available time available staff)/ (average time to prepare a test case),How many test cases do we need?,Cost (available budget) / (average preparation cost per test case),Select the minimum number of the two,43,Example,Development budget: $4 million, %10 of which spent on preparing test cases Each test case costs $250 or 4 hours The duration of project is set to be 25 weeks (of 40 hours each) and 5 staffs for test cases How many test cases should be prepared?,From cost N1 = (4,000,000 0.1) / 250 = 1,600 From time N2 = (25 40 5) / 4 = 1,250 N = min (N1 , N2), therefore, N = 1,250,44,Allocating Test Time,How to allocate test time among system components (e.g., acquired components, developed components) test type (e.g., unit test, integrated test, system test) and,45,Test Time: Systems /1,Allocate less than 10% of the remaining time to certification test of acquired components Allocate time to the developed components based on their significance,OS, System software,Developed components,Acquired components,Interface to other systems,46,Test Time: Systems /2,Example: Total test time: 340 h Test case proportion: Associated system components: 64.7% Operating system component: 23.5%,Interface to other systems (40 h),Acquired components (20 h),Developed components (200 h),OS, System software (80 h),Associated System Components 220 h,47,Test Case Invocation,In what order the system should be tested? Recommended sequence of system test: Acquired components (certification test only) Developed product Other systems, OS, etc. (load test only),48,Section 3 Test Coverage Metrics,49,Test Coverage Metrics /1,Coverage of what? Statement coverage Branch coverage Component/Module coverage Specification coverage GUI coverage ,50,Test Coverage /1,Statement coverage (CVs) Portion of the statements tested by at least one test case.,51,Test Coverage /2,Branch coverage (CVb) Portion of the branches in the program tested by at least one test case.,52,Test Coverage /3,Component coverage (CVcm) Portion of the components in the software covered and tested by at least one test case,53,Test Coverage /4,GUI coverage (CVGUI) Portion of the GUI elements (e.g., menus, buttons, multiple selections, text fields, etc.) in the software covered and tested by at least one test case.,54,Test Pass, Failure & Pending Rate/1,Test Pass Rate (Rtp) Portion of the test cases that were executed successfully (i.e., produced expected output).,55,Test Pass, Failure & Pending Rate/2,Test Failure Rate (Rtf) Portion of the test cases that were not executed successfully