翻译以.原文和在同一文件中前

软件工程德克萨斯州奥斯汀市78712亚当·波特(AdamA.Porter)：马里兰大学计算机科学专马里兰大学帕克分校劳伦斯·沃塔(LawrenceG.Votta)：摩托罗拉公司伊利诺伊州阿灵顿高地，1但是在软件工程的研究中，并不成功。与它们在其他科学领域的广泛运用我们认为这些文章多以实现为导向，也就是说它们认为运用最大的在于实比如，NormanFenton等人[1]许多的数据设计很失败，不能与大系统相匹配，而且实施时间过短。VictorBasil[2]。PhilipJohnson也提到实践者被测试或量化[3]诚然，这样那样的原因影响着的运用，但是我们相信即使所有这些问题都我们尝试用改变创建软件方式的经验让我们相信要达到目的，必须还要以在试着提高朗讯开发系统里的软件检验流程中，我们得出这一结论，最大的不2、为什么用们的理论。所以有以下步骤：3、的状前面提到是将我们相信的运用到我们所观察到的事物上。理想情况下，这目前的优的平均质量进步很大。现在研究普遍受过良好的教育，懂得程的[4]。我们已经和目前活跃的学者有过多次交谈，这些人已对或开始对感。当然，的形因有很多。很多研究和实践者已经解决了提高应有一些有、被广泛的文章试着提高我们对现状的关注。Tichy等人[5]以及Wallace和Zelkowitz[6]均认为相对于工程学的其他领域，在软件工的，Kitchenham和Pfleeger为ACMSigSoftSoftwareEngineeringNotes写过一系列文章，于行业数据、对行业经验有重要且详细记录的。这一的先驱包括国家航天局的软件工程，计算机科学公司和马里兰大学[7]。体系上的尽管有上述优势，但问题依然严重。这源于对是什么和为什么做 通常当人们说我们需要做软件工程的时，他们的意思是研究结果应该举个例子，在编译程序最优化研究中，已经辨认了通用代码使用模式。例分、的用。太多的关注一些显而易见的东西。但有些明显的东西并非如此，所以我们么靠所做的论证错在哪儿了？我们清楚地认识到正确的事情并非是上明显的，许多的唯一卖点就是大量的数据，但仅仅有数据是远远不够的。仅仅展示数据问题更基本的一面是很多缺乏假设。研究者没有提出问题，研究没有明确4、未来不仅是，所有研究的目标都是提高研究和实践的现状。如果我们想让实证创造更好的研究很少是明确的。除了用一个研究试着解决一些大问题，须做多个研帮助他人我们结论的方法。可信的解 设计一个5、的结关于的文章也应讨论它们，这些成分即：研究背经做了哪些研究，还有哪些问题尚待回答，以及关注哪些问题。假假设是必不可少的。它们陈述了我们研究问题。有时人们对这一概念并不清组找出问题”。实验设效度的因数据分析和展假设检验决定原假设可被的置信水平。置信水平是原假设被错误的可能性定性分析则是运用量化程度不高的数据，如观察、采访、等。当我们要理解人[8。软件工程研究中，定量分析要比定性分析用得多，但我们在未来会看到的定性来检验定型数据的，两者是互补品、用来相互验证、是同一里不同形式的数据”。结果和结讨论结果的实际意义。如果结果是普遍通用的，管理者和开发能怎样用它？确保你提供了足够的信息让他人研究。6、具体步设计研相反，须缩小问题，问能引出重要答案的问题。Knight和Leveson对N-VersionProgramming的研究是一个说明问题的好例子[9]靠度可能没有那么高。相互关联的，进而N-VersionProgramming不能实现它有关高可靠度的承诺。，的不是所有问题都能像N-VersionProgramming一样引出一个单一的。对于很们那么有用。些研究，减轻开发的负担。这是一个很有力的优势，因为我们行业伙伴不想参知识来解决，这样吸纳软件工程专业以外的将十分有用。码的一个例子来自于基于朗讯科技公司的代码项目（CodeDecayProject）[11]。机构的历史、发展政策和编程标准。这个项目的目标是定义反应变量，证码的 获取数我们尚未关注的一个资源是版本控制系统（VCS。很多对不同流程和工具的长期影响的分析依赖于重建软件在不同时间点的片段的能力。版本控制系统研发者对系统做出的每一个改变，作为结果，可以重建续的片段。版本控制系统的例子包括修订控制系统（RCS）[13]和源代码控制系统（SCCS）[14]中发生了什么，有时比研发的更可靠。同时，版本控制系统的数据经得起自动望将模拟和建模结合起来指导研究。一个有趣的例子是Solheim和Rowland[15]做的有关系统的研究。其中，其他例子包括用数学建模检验更改某些性的成本收益[16]，用实验设计理论产生测试囊括其教育这种教学模块可以以教育的形式展开，教育作为自然科学教育的一个。要建立这样的，研究者要把打包进一本手册，手册里包含讲7本文提供了关于目前的概述，了它的优势和不足，讨论了在创造为软要改进现状，须有更好的设计，并从中汲取可靠的解释。作为未来结尾：设计更好的研究，获取数据，在中获得他人的帮助。尽管与其他科学和工程学领域相比，实证科学尚不成熟，但在进步，我们8、参考文[1]N.Fenton,S.L.Pfleeger,andR.Glass,ScienceandSubstance:AChallengetoSoftwareEngineers.IEEESoftware,1994.11(4):p.86-95.[2].V.Basili,Editorial.EmpiricalSoftwareEngineeringJournal,1996.[3].P.M.Johnson,ProjectLEAP:Lightweight,Empirical,asurementdysfunction,andPortableSoftwareDeveloperImprovement,inDepartmentofInformationandComputerSciences.1997,UniversityofHawaii,Honolulu.[4].D.Pregibon,etal.,StatisticalSoftwareEngineering,.1996,NationalAcademyofSciences:Washington,D.C.[5].W.F.Tichy,P.Lukowicz,L.Prechelt,andE.A.Heinz,ExperimentalEvaluationinComputerScience:AtativeStudy.JournalofSystemsandSoftware,1995.28(1):p.[6].M.V.ZelkowitzandD.Wallace,ExperimentalvalidationinsoftwareInformationandSoftwareTechnology,1997.39(11):p.735-[7].V.R.Basili,etal.TheSoftwareEngineeringLaboratory--AnOperationalSoftwareExperienceFactory.in14thInternationalConferenceonSoftwareEngineering.1992.Melbourne,Australia.[8].B.GlasserandA.Strauss,Thediscoveryofgroundedtheory:Strategiesforqualitativeresearch.1977,Chicago:AldinePublishing.[9].J.KnightandN.Leveson,AnExperimentalEvaluationoftheAssumptionofIndependenceinMulti-VersionProgramming.IEEETransactionsonSoftwareEngineering,1986.SE-12(1):p.96-109.[10].B.Schneiderman,R.Mayer,D.McKay,andP.er,ExperimentalInvestigationsoftheUtilityofdetailedFlowchartsinProgramming.CommunicationsoftheACM,1977.20(6):p.373-381.[11].S.G.Eick,etal.,DoesCodeDecay?AssessingtheEvidencefromChangeManagementData.IEEETransactionsonSoftwareEngineering,(toappear).[12].C.M.Judd,E.R.Smith,andL.H.Kidder,ResearethodsinSocialRelations.1991,FortWorth,TX:Holt,RinehartandWinston,Inc.[13].W.F.Tichy,Design,Implementation,andEvaluationofaRevisionControlSystem,inProceedingsoftheSixthInternationalConferenceonSoftwareEngineering.1982:Tokyo,Japan.p.58—67.[14].M.J.Rochkind,TheSourceCodeControlSystem.{IEEE}TransactionsonSoftwareEngineering,1975.1(4):p.364—370.[15].J.A.SolheimandJ.H.Rowland,AnEmpiricalStudyofTestingandIntegrationStrategiesUsingArtificialSoftwareSystems.IEEETransactionsonSoftwareEngineering,1993.19(10):p.941-949.[16].W.Harrison.Change-ProneModules,LimitedResources,andMaintenance.inwess.1996.Monterey,CA.[17].S.R.DalalandC.L.Mallows,Factor-coveringdesignsfortestingTechnometrics,1998.40:p.234-[18].G.V.Glass,B.McGaw,andM.L.Smith,Meta vsisinsocialresearch.1981,BeverlyHills,CA:Sage.[19].A.A.PorterandP.M.Johnson,AssessingSoftwareReviewMeetings:ResultsofaComparativeysisofTwoExperimentalStudies.IEEETransactionsonSoftwareEngineering,1997.23(3):p.129-145.EmpiricalStudiesofSoftwareEngineering:ADewayneE.AdamA.LawrenceG.ElectricalandComputerComputerUniversityofTexasatUniversityof1501W.ShureAustin,TXCollegePark,MDArlingtonHeights,ILInthisarticlewesummarizethestrengthsandweaknessesofempiricalresearchinsoftwareengineering.Wearguethatinordertoimprovethecurrentsituationwemustcreatebetterstudiesanddrawmorecredibleinterpretationsfromthem.Wefinallypresentaforthisimprovement,whichincludesageneralstructureforsoftwareempiricalstudiesandconcretestepsforachievingthesegoals:designingbetterstudies,collectingdatamoreeffectively,andinvolvingothersinourempiricalenterprises.EmpiricalStudies,SoftwareAnempiricalstudyisreallyjustatestthatcompareswhatwebelievetowhatweobserve.Nevertheless,suchtests,whenwiselyconstructedandexecutedandwhenusedtosupportthescientificmethod,playafundamentalroleinmodernscience.Specifically,theyhelpusunderstandhowandwhythingswork,andallowustousethisunderstandingtomateriallyalterourworld.Yetinsoftwareengineeringresearch,empiricalstudieshavenothadthesamesuccess.Thisseemsoddgiventheirwideuseinothersciences.Thisproblemhasbeenwidelydiscussedandmanyarticleshavepointedoutpossiblecauses.Weargue,however,thatmanyofthesearticlesare“implementation-oriented”.Thatis,theysuggestthatthebiggestbarrierstousingempiricalstudieslieinthedetailsofconductingthem.Forexample,NormanFentonetal.[1]pointoutthatmanyempiricalstudieshavepoorstatisticaldesigns,don’tscaleuptolargesystems,andareconductedovertooshortatime.VictorBasili[2]suggeststhatthemanydifferences

betweenindividualsoftwareprojectsmakecomparisondifficult.PhilipJohnsonalsoremarksthatpractitionersmayresistbeingmeasured.[3].Surely,theseandmanyotherfactorsaffecttheuseofempiricalstudies.Nevertheless,webelievethatevenifalltheseissuesdisappeared,empiricalstudieswouldstillfailtohavetheimpacttheyhavehadinotherfields.Thisisbecausethereisagapbetweenthestudiesweactuallydoandthegoalswewantthosestudiestoachieve.Ourexperienceinattemptingtouseempiricalstudiestochangehowadevelopmentgroupbuildssoftwarehasconvincedusthatwemustalsotakea“requirements-oriented”view.Thatis,thatwemustthinkharderaboutwhatexperimentsreallyareandhowtheycanbemosteffectivelyusedtoimprovesoftwaredevelopment.WecametothisconclusionwhiletryingtoimprovethesoftwareinspectionprocessusedinaLucentdevelopmentsetting.Wefoundthatourgreatestdifficultieswerenotindesigningandconductingindividualstudies(whichwasbynomeanseasy).Ourgreatestdifficultieswereinconceptualizingandorganizingabodyofworkthatcouldbereliedonasthebasisforchanginganorganization’slong-practiceddevelopmentprocesses.Moreover,webelievethatthisproblem–definingandexecutingstudiesthatchangehowsoftwaredevelopmentisdone-isthegreatestchallengefacingempiricalresearchers.Therefore,inthisessaywewillexaminethenatureandpurposeofempiricalstudies,discusshowtheyarecurrentlyused,andoffersomesuggestionsforimprovingtheminthefuture.WHYEMPIRICALAlllargesoftwareprojectsfollowsomeunderlyingdevelopmentprocessthatincludesstagessuchasrequirementsdefinition,functionaldesign,unitimplementation,integration,andsoon.Thewayinwhichthesestagesareconducted,thetoolsthatareusedtosupportthemandtherationaleforngso,however,varieswidely.Somecompanieshaverigidprocessesthatallprojectsfollow.Othersallowindividualmanagerstomakedecisionsbasedontheir alexpertise.Otherssimplyfollowinstitutionaltraditionsforlackofsuitablealternatives.Nomatterwhichapproachistaken,inalmostallcases,thereislittlehardevidencetoinformthesedecisions,andtheircostsandbenefitsarerarelyunderstood.OnereasonforthisisthatsoftwareengineeringresearchhasfailedtoproducethedeepmodelsyticaltoolsthatarecommoninotherThesituationindicatesaseriousproblemwithresearchandpracticeinsoftwareengineering.Wedon’tknowthefundamentalmechanismsthatdrivethecostsandbenefitsofsoftwaretoolsandmethods.Withoutthisinformation,wecan’tlwhetherwearebasingouractionsonfaultyassumptions,evaluatingnewmethodsproperly,orinadvertentlyfocusingonloyoffimprovements.Infact,unlessweunderstandthespecificfactorsthatcausetoolsandmethodstobemoreorlesscost-effective,thedevelopmentanduseofaparticulartechnologywillessentiallybearandomact.Empiricalstudiesareakeywaytogetthisinformationandmovetowardswell-foundedEmpiricalstudiestakemanyforms.Theyarerealizednotonlyasformalexperiments,butalsoascasestudies,surveys,andprototyexercisesaswell.Nomatterwhatitsformis,theessenceofanempiricalstudyistheattempttolearnsomethingusefulbycomparingtheorytorealityandtoimproveourtheoriesasaresult.Therefore,empiricalstudiesinvolvethefollowingsteps:formulatinganhypothesisorquestiontoobservingaingobservationsintoyzingthedata,drawingconclusionswithrespecttothetestedOfthese,thelaststep–drawingconclusions-isthemostimportantandtoooftentheleastwelldone.It’simportantbecauseit’sherethatwegettheinformationthatwillenableustoguide,tochangeandtopushourfield.It’sherethatwepinpointinefficiencies,identifywherelargeimprovementscanbemade,anddeterminewhetherourstill-formingideasareon-track.It’sthereasonwhywedoempiricalstudies.Theothersteps,howeverindispensable,areonlyprologue.Ofcourse, ngallofthesestepswellisdifficult.Donewell,however,thepayoffswillbelarge,includingthat:knowledgeisencodedmore yofforerroneousresearchideasarediscarded

high-payoffareasarerecognizedandcorrectlyvalued,importantpracticalissuesareTHESTATEOFEMPIRICALWehavesaidthatempiricalstudiesareusedtocomparewhatwebelievetowhatwesee.Ideally,thesetestsshouldallowustopositivelyaffectthepracticeofsoftwaredevelopment.Inthissectionwewillexploretowhatdegreewe,asaresearchcommunity,arelivinguptothisideal.CurrentEmpiricalsoftwareengineeringhasmaturedconsiderablyoverthelast10-20years.Considerforexample:Insomesoftwareengineeringsub-fieldsempiricalvalidationisconsidered,ifnotastandardpart,thenapowerfuladditiontoresearchpapers.Thishasbeenespeciallynotableinthetestingcommunity.Thequalityoftheaverageempiricalstudyisrising.Researchersare ingbettereducatedaboutempiricalstudiesandhowtoconductthem.Consequently,weareseeingincreasinglymorecomprehensivestudiesconductedonincreasinglyrealisticprogramsandprocesses.Fundingagenciesarerecognizingthevalueofempiricalstudies.IntheU.S.forexample,NationalScienceFoundation(NSF)programssuchastheExperimentalandIntegrativeActivitiesprogramsupportsresearchwithadecidedlyexperimentalflavor.TherecentlyproposedInformationTechnologyResearch(ITR)programalsostressesthatproposalsincludeastrongvalidationcomponent.OtherexamplesincludeNationalAcademyofSciencessponsoredworkshoponthetopicofstatisticsandsoftwareengineering[4].We’vehadmanytalkswithcurrentlyactiveresearcherswhohave einterestedinandarebeginningtodoempiricalstudies.Andfinally,therehavebeenseveralempiricalstudies-relatedtutorials,panelsandstate-of-theartpresentationsatmajorsoftwareengineeringconferencessuchasICSE,FSE,ICSMandothers.Ofcoursemanyfactorscontributetothissituation.Manyresearchersandpractitionershavetackledtheproblemofincreasingtheuseandeffectivenessofempiricalstudies.Forexample:Therehavebeenseveralinfluentialandwidelyquotedarticlesattemptingtoraiseourconsciousnessaboutthestateofempiricalstudiesinsoftwareengineering.Tichyetal.[5]andWallaceandZelkowitz[6]botharguethatempiricalstudiesareunderusedinsoftwareengineeringrelativetootherareasofengineering.Bothferociouslycondemnsoftwareengineeringresearchersfornotvalidatingtheirresearchideasandbothhavebeeninvaluablemakingthisahighprofileissue.Thereisagrowingawarenessthatsoftwareengineeringresearchersmustbeeducatedaboutconductingempiricalstudies.Tothisend,KitchenhamandPfleegerwroteaseriesofarticlesforACMSigSoftSoftwareEngineeringNotes.Thesearticlescoveredavarietyoftopicsincludingthelogicalfoundationsanddesignofempiricalstudies,theiroperation,andtechniquesforcollecting,yzingandinterpretingdata.Severalresearchgroupswereinstrumentalinincreasingresearcheraccesstoindustrialdata.Todaywefindmanypaperswithsignificant,detailedaccountsofindustrialexperiencebasedonindustrialdata.OneoftheforerunnersofthisapproachwastheSoftwareEngineeringLaboratoryofNASA,theComputerSciencesCorporation,andtheUniversityofMaryland[7].Finally,manyfineresearchershavewadedinanddonetheirownempiricalstudies.SystemicDespite,ormaybebecauseof,thestrengthslistedabovetherearesomeseriousproblems.Thesestemfrommisunderstandingsaboutwhatempiricalstudiesareandwhywedothem.Beforewecanimproveouruseofempiricalstudieswehavetoeliminatesomeproblematicpracticesandbeliefs.Oftenwhensomeonesaysthatweneedmoreempiricalstudiesinsoftwareengineering,theyreallymeanthatresearchresultsshouldbeempiricallyvalidated.Theywantresearcherstodemonstratethevalueoftheirnewideasasearlyaspossible.Thisisagoodideaformanyreasons.Webelieve,however,thatitisimportanttorememberthatempiricalstudiescanbeusednotonlyretrospectivelytovalidateideasafterthey’vebeencreated,butalsoproactivelytodirectourresearch.Forexample,incompileroptimizationresearchempiricalstudieshaveidentifiedcommoncodeusagepatterns.Knowing,forinstance,thatbranchingbehaviorisnotusuallyrandom,helpsidentifyandjustifythepotentialvalueofresearchonbranchprediction,aggressivepre-fetching,etc.Inshort,weshoulduseempiricalstudiesalsotodriveourresearchInprogramcommitteemeetingsweoftenhearlengthydiscussionsovertheexactstatisticaltestsusedinastudyorwhetheritwouldn’thavebeenbettertohavedoneonethingoranother.Thesediscussionsreflectavainsearchfortheperfectstudy.Well,we’vedonemanystudiesandwe’veneverdoneoneperfectly!Ofcourse,wewanttoseeproperstatisticsused.Butaswewilldiscussshortly,what’simportantisnotwhetherthestudyistextbookperfect,butwhetherthestudyanditsconclusionstakenasawholeareToomanyempiricalstudiesstudytheobvious.Asthissometimesshowsthattheobviousisn’tsoobvious,we

wouldn’tdiscourageanyonefromngsuchwork.Nevertheless,itmakesuswonder,“ifempiricalstudiesmostlyjustconfirmtheintuitivelyobvious,thenwhat’swrongwithargumentbyintuition”.Clearly,webelievethattherearethingsthataretrue,butthatarenotintuitivelyobvious.Furthermore,webelievethatsomeofthesefindingswillbevaluabletosoftwareresearchandpractice.Therefore,weneedtothinkmuchharderaboutthequestionswearestudyingempirically.Therearetoomanypaperswhoseonlysellingpointisthattheyhavelotsofdata.Dataisnotenough.Justpresentingdataorsimplyapplyingcurve-fittingalgorithmstothemmaybeuseful.Buttheydon’tusuallyhelpusunderstandwhythedataisasitis.Ourdatashouldbeusedtoanswerquestions,notjusttofillgraphs.Amorefundamentalaspectofthisproblemisthatmanyempiricalstudiessimplylackhypotheses.Theyposenoquestions,theyservenowell-definedend.Thusatofthestudytheresearchercanonlypresentobservationsaboutthedata.Allstudies,evencasestudies,shouldbedesignedtoanswersomequestion.Aswesaidearlier,themostimportantpartofnganempiricalstudyisdrawingconclusions.Manypapersfailtodoanythingwiththeirresults.WeneedtolearnsomethingfromeverystudyandrelatethesethingstotheoryandSincemanyresearchersarereluctanttodrawconclusionsfromtheirdata,it’seasytoimaginethattheyaren’ttoohappytogeneralizethemeither.Insteadofspeakingthoughtfullyabouttheirworktheycloaktheresultsin“weaselwords”.Somuchsothat,often,intheend,theysaynothing.There’sobviouslyabalancetobereachedherebecausewedon’twantresearcherstoover-generalize,Butontheotherhand,ifwecan’tdiscusswhatastudy’sresultsmightmeanthenit’shardtomakeprogress. Thegoalofallresearch,notjustempiricalstudies,istoimprovethestateofresearchandpractice.Ifwewanttoempiricalstudiestoimprovesoftwareengineeringresearchandpractice,thentherearetwothingsthatweneedtodobetterinthefuture.Saidsimply,weneedtocreatebetterstudiesandweneedtodrawmorecredibleconclusionsfromthem.CreatingBetterEmpiricalCreatingbetterstudiesmeansngstudiesthathavesomechanceofdirectingourresearch.Itimpliesthatwemustbeclearaboutthegoalsofourstudies,designthemmoreeffectively,and izetheinformationwegetoutofTodothisweshouldconsideratleastthefollowingOurstudiesshouldstrivetoestablishprinciplesthatarecausal,actionableandgeneral.ForafactorAtocause eBit’snecessarythatAandBarecorrelated,thatAprecedesBintimeandthatthereisaconstructive,testabletheoryexplaininghowAaffectsB.WithoutcausalityyouhavenoabilitytocontrolyourAprincipleisactionableifthecausalagentAcanbeeffectivelycontrolled.Forexample,knowingthatlargersystemsnormallyhavemorebugsmaynotbeanactionableprincipleifthedevelopercan’tmakethesystemsmaller.Theprinciplesshouldbeapplicableinaswideavarietyofcircumstancesaspossible.Whenwehaveacausalrelationshipweknowwhysomethinghappens.Ifntisactionable,thenwehaveaknobthatcanbeturnedtocontrolthe e.Ifitisgeneralitwillbeusefultoawiderangeofpeopleinawidesetofcontexts.Ourstudiesshouldtrytoaddressimportantquestions.Therearemanyquestionstoanswer.Answeringsomeofthemwillbecheaperthanansweringothers;usingthoseanswerswillhavemoresignificanceinsomecasesthaninothers.Thisconsiderationimpliesthatweneedtospendagooddealoftimeunderstandingwhywe’rengourstudiesandwhatresultsmightcomefromthem.Individualstudiesarerarely,ifever,unequivocal.Insteadoftryingtosolvelargeissueswithasinglestudywemustattackitwithseveral;eachexaminingdifferent,butcomplementaryaspects.Herethecriticalissueistouseeachnewstudytogenerateandrefineourhypotheses.Empiricalstudiesareexpensiveandtaketime.Ifwemustdomultiplestudies,thenwehavetofindwaystogettheinformationweneedatalowcost.Thismayalsomeanthatwehavetotakesomeshortcutsinourexperimentaldesignsortacklesmaller,morefocusedproblems.Wewillalsoneedtoenlistthehelpofothers.Empiricalstudiesgaincredibilitywhentheyareredoneandrechecked.Weneedtofindwaystohelpotherstoreproduceourresults.CredibleThecredibilityofastudyreferstothedegreeofconfidencewehaveinitsconclusions.Ifstudiesaren’tcredible,thenthetimespentngthemwaswasted.Toimprovethecredibilityofourstudieswemustconsiderseveralissues.Ifwearetryingtoestablishtheexistenceofcausalrelationships,weneedtodesignexperimentswithhighvalidity.Validity,aswewillexplainlater,isacharacteristicofanempiricalstudyandisthebasisofestablishingcredibleconclusions.Therearethreetypesofvaliditythatareparticularlyimportant:internal,external,andconstructvalidity.

Ourstudies(nomatterhowtheyaredone)shouldalwayshavehypotheses.Witheverystudywemustdefinewhatwearecomparingandwhy.Oftenastudywon’tbepowerfulenoughtoshowacausalrelationship.Still,inmanycaseswecanpositseveralalternativeexplanationsforthedataandthenuseotherdatatodiscreditthem.Thisstilldoesn’tshowcausality,butitcanatleastremoveobviousalternativeexplanationsfromWeshouldavoidthetemptationtomeasureeverythingtothefinestpossibleprecision.Sometimesitwillbeenoughtoidentifyanupperandlowerbound;othertimesitwillbeenoughtomeasureatagrossresolution.Thedefinitionofadequateprecisionwilldependontheproblem,butusingcoarsemeasurementsmaybeonewaytolimitstudycosts,whilestillgettingimportantinformation.Ourdataandproceduresneedtobemadepublicsothatotherscanunderstand,yzeandpossiblyreplicateourstudies.Frankly,thiscanbereallydifficult,andwehaven’talwaysmanagedtokeepupourselves,butwebelieveit’sworththeeffort.Inourcareerswe’vedesignedandconductedanumberofstudies.Nonehavebeenwithoutflaws.Ourconclusionisthatnostudyisperfectandthattherealchallengeistocreate,designandconducthigh-impact,credibleThisinvolvesmanagingtrade-offsinsuchawaythatweaccuracyofinterpretation-theresultsweseearenotreallytheresultofsomeunknowninfluence,relevance-ourresults lussomethingimportantaboutsoftwareengineering,andimpact-ourresultsaffectthepracticeoforresearchintosoftwareengineeringsubjectresourceconstraints-studiesareexpensive;wemustworkwithinresourcelimitations,andrisk-studies,especiallythosedoneinindustry,candisruptorputatriskindustrialpartners;wemustminimizetheseproblems.THESTRUCTUREOFANEMPIRICALInthissectionwediscussthestructureandcomponentsofempiricalstudies.Weexpectthatgoodempiricalstudieswillhaveeachofthesecomponentsandthatpaperswrittenaboutthestudieswilldiscussthemaswell.Thesecomponentsare:researchthreatsto ysisandpresentation,resultsandResearchAllstudiesfocusonaproblem.Heretheproblemisdefinedanditsterminologyexplained.Thissectionlinksthestudygoalstowhat’scurrentlyunderstoodabouttheproblem.Thissectionhastwoparts.ProblemDefinition:Wedefinetheproblemandexplainit'simportantterminology.ResearchReview.Weprovidethehistoricalcontextsurroundingtheproblem.Wedescribewhatweknowabouttheproblem,whathasbeendonepreviously,whatquestionsstillremaintobeansweredandwhatquestionswillwebefocusingon.Hypothesesareessential.Theystatetheresearchquestionsweareasking.Sometimesthereisconfusionsurroundingthetermhypothesis.Infacttherearereallytwokindsofhypotheses.Thetrickistothinkofastudyasaprocedureformakingacomparison.Therefore,westartatwithhigh- questionsandrefinethemintolow-level,concretequestions.hypothesesarehigh-level,naturallanguagestatementsthatareusuallystatedineverydayterms.Theysaythingslike,“meetingsareanindispensablepartoftheinspectionprocess”.Concretehypothesesarestatedintermsofthestudy’sdesign.Theymaysaythingslike,“teamswhodoinspectionswithmeetingsfindmoredefectsthanteamswhonspectionswithoutthem.”Webeginbystatingourhypothesesfirstineverydayterms.Thenwetranslatethemtotermsthatexistinthestudy’sdesign.Tothedegreethatthismapisdonewell,comparisonsmadeatthelevelofconcretehypothesescanbemappedbacktothecomparisonsmadeatthelevelStudyAstudy’sdesignisadetailedplanforcreatingthedatathatwillbeusedtotestitshypotheses.IthasseveralOnecomponentisasetofvariablesthatlinkcausesandeffects.Typically,therearetwokindsofvariables:dependentandindependent.Independentvariablesareattributesthatdefinethestudysetting.Insomecases,especiallywhencomparingtwosituations,thesevariablesareactivelymanipulated.

Dependentvariablesareend-processoutputswhosevaluesareexpectedtovarypredictablywhenthevaluesofindependentvariableschange.Thestudydesignmayalsoincludeaplanforsystematicallymanipulatingtheindependentvariableswhileobservingthedependentvariables.Thefinalcomponentistheoperationalcontextofthestudy.Thisisadescriptionofthephysical,inlectualandculturalsurroundingsinwhichthestudytakesplace.Itisincludedsothatthestudy’suserscanbetterinterprettheThreatstoThreatstovalidityareinfluencesthatmaylimitourabilitytointerpretordrawconclusionsfromthestudy’sdata.Thereareatleastthreekindsofvaliditythatmustbeprotectedfromsuchthreats.Constructvaliditymeansthattheindependentand accuray Internalvaliditymeansthatchangesinthedependentvariablescanbesafelyattributedtochangesintheindependentvariables.Externalvaliditymeansthatthestudy’sresultsgeneralizetosettingsoutsidethestudy.DataysisandTwogeneralapproachestopresentingandyzingdataarecalledtativeandQualitativeysis.tativeyses,asthenamesuggests,dealmainlywithcomparingnumericdata.Thecomparisonsaretypicallyaimedatrejectingornotrejectinganullhypothesis.Twoofthetoolsusedintativeysisarehypothesistestingandpowerysis.Hypothesistestingdeterminestheconfidencelevelatwhichthenullhypothesiscanberejected.Theconfidencelevelisameasureoftheprobabilitythatthenullhypothesiswillbeerroneouslyrejected.Somepeoplebelievethatthisconfidencelevelmustbelessthan1in20or0.05foraresulttobesignificant.Itdoesn’thavetobe.Insituationswheredataisplentifulandmeasurementsprecise,higherconfidencelevelsmaybecalledfor.Sincedataisoftenlimitedandmeasurementimpreciseinstudiesofsoftwareengineering,lowerconfidencelevelsmaybejustified.Inanyevent,wesuggestthatresearchersreportth