empirical evaluation in end-user software engineering在最终用户软件工程的实证评价

Empirical Evaluation in End-User Software Engineering,Janice SingerNational Research Council Canada,Singer WEUSE IV,2,Outline,Summary of papers wrt empirical evaluation looking at:Research QuestionDomainMethodSubjects/Objects of studyResultsThemes/cross-cutting issuesQuestions for Discussion,Singer WEUSE IV,3,Spreadsheet debugging behaviour of expert and novice end-users - Bishop, McDaid,Research Question4 Basic RQ concerning performance of expert vs. novice users in detecting and correcting errors, debugging behaviour, and cell inspection coverageDomainSpreadsheetMethodExperiment, Qualitative inquirySubjects/Objects of study13 professionals and 34 accounting and finance students (experts and novices),Singer WEUSE IV,4,Results,Experts perform better than novices at detecting errors that require deep understandingCell coverage correlates with performance - experts look at more cells than novicesThere is a specific pattern of cell inspection depending on the characteristics and place of the cell in the spreadsheetA tool whose aim was to increase cell inspection coverage showed a trend, but did not significantly improve performance.,Singer WEUSE IV,5,Gender in EUSE - Burnett, et al.,Research QuestionAre the strategies employed by male and female EUSE in debugging different?DomainSpreadsheetsMethodExperiment, qualitative studySubjects/Objects of studyMales, females, professionals, students,Singer WEUSE IV,6,Results,There are significant gender differences in strategies for approaching testing and debuggingSome of the strategies preferred by females are not well supported in end-user environmentsModeling of problem solving behaviour may improve females confidence, and therefore their performance on tasksGender matters,Singer WEUSE IV,7,End users as unwitting software developers - Costabile et al.,Research QuestionAre our theories/descriptions accurateDomainGroup of companies who cooperate in candy distributionMethodNot clearSubjects/Objects of studyUsers of a web-based portal,Singer WEUSE IV,8,Results,In industrial practice, we see a variety of end-users with their own attendant needs.,Singer WEUSE IV,9,An EU oriented graph based visualization for spreadsheets - Kankuzi, Ayalew.,Research QuestionIs our tool technically correct?DomainSpreadsheetMethodGenerate and test ?Subjects/Objects of studyGenerated visualizations,Singer WEUSE IV,10,Results,For the most part, the algorithm that drives the visualization is correctNeed to evaluate whether the visualization is actually useable by and useful to end-users,Singer WEUSE IV,11,Using two heads in practice - Karlsson,Research QuestionAre dyads more effective than singletons (or nominal dyads) in debugging spreadsheets? Is there a process loss for two people working together?DomainspreadsheetsMethodField experiment - experiment conducted in the fieldSubjects/Objects of studyProfessionals,Singer WEUSE IV,12,Results,Dyads make fewer spreadsheet development errors than monadsThere is no significant difference in performance between nominal dyads and real dyads - therefore unable to determine whether there is a process loss or not,Singer WEUSE IV,13,TDD: can it work for spreadsheets? McDaid, Rust, Bishop,Research QuestionIs test-driven development an appropriate process for spreadsheet development?DomainspreadsheetMethodDeveloped tool to support TDD then case studies (real professionals using tool to develop spreadsheets)Subjects/Objects of studyProfessionals,Singer WEUSE IV,14,Results,TDSD easy to understand and useDevelopment time increaseOverall, participants seemed to believe TDSD effective in reducing errorsSome improvements to the tool suggested,Singer WEUSE IV,15,Software support for building EUP environments in the automation domain - Prhofer, et al.,Research QuestionIs our solution technically correct/feasible?DomainAutomationMethodCase study, Reimplementation of existing systemsSubjects/Objects of studyDevelopers of system/Two existing systems,Singer WEUSE IV,16,Results,Existing systems were able to be implemented in framework, with a great reduction in the code size,Singer WEUSE IV,17,Patterns in mash-ups - Wong, Hong,Research QuestionAre there typical application domains for mash-ups? DomainWeb programming/mash-upsMethodSurvey (in the sense of categorization of) of and qualitative analysis mash-upsSubjects/Objects of studyPopular GreaseMonkey scripts and 22 recommended mash-ups,Singer WEUSE IV,18,Results,Mashups can be categorized according to their functionality. These patterns include personalization, search, aggregation amongst others,Singer WEUSE IV,19,Summary of empirical studies,Wide variety of research questionsNot so much use of theory (is it necessary?)Not so wide variety of domainMostly spreadsheetWide variety of methodsExperiments, surveys, case studies, tool correctnessSubjects/Objects of studyVaried and related to research question,Singer WEUSE IV,20,Themes and Questions -,THEME: DomainWhat other domains should we be looking at in terms of empirical evaluation or tool support?Related to this, shouldnt we be doing more qualitative and observational work in real settings?,Singer WEUSE IV,21,THEME: End-User CharacteristicsWhat other end-user characteristics do we need to be aware of when studying and designing tools for end-users?Are there general cognitive limitations in terms of abilities or is it mostly poor tool support that limits the ability of end-users,Singer WEUSE IV,22,THEME: Technical CorrectnessHow do we help end-users test, debug, determine the technical correctness of their solutions?What methods can we use to test the technical correctness/quality of our solutions - can experimentation with humans alone do this?,Singer WEUSE IV,23,THEME: Software EngineeringWhat can studies in EUSE tell us about SE, and vice versa?Given the changing world of SE (e.g., SOA, component and model based development, interoperability and integration issues), is there any longer a difference between EUSE and SE?,Singer WEUSE IV,24,THEME: Building the Research AreaWhat are the critical/BIG research questions?Is there enough information to start building meta-theories, do meta-analyses?,Singer WEUSE IV,25,Discussion,Already using theories, but many of them are implicit, and what we need to do is be explicit about them.Concept of theories have many meanings - but how can we generalize findings from individual cases to more general findings. So many of our findings are context dependent, it is very difficult to generalize to other contexts. So, in this case, how do we transfer knowledge. But cant work with simply theories.Do we really need qualitative studies first - yes, they can provide much information. Who are end-users? Can we characterize them. There is a lot of diversity and those points of diversity matter in terms of how to help people.,Singer WEUSE IV,26,In EUSE, there is a lot of qualitative work, and highly regarded.Question bears on difference significance and meaning. Ask about two classes - e.g., male vs. female. But really there is a huge overlap in the two populations. But changing for one population will often help the other. Is it true for experiments reported here. Can tools for EUSE benefit software engineers.Huge individual differences - no typical male and no typical female. So when find differences, really, what finding is a barrier that affects performance. Statistics lie - must accept that the numbers only make sense in particular situations. Must be explicit about what the numbers mean, how we apply the generalizations to discover meaningExternal validity - even if you create a tool that is useable in a lab experiment and performs well according to a set of tasks - it doesnt mean that the general effect is there. Is it possible to run field studies by releasing the tool and seeing if it works in context.,Singer WEUSE IV,27,But difficult because if you fail, you cant always be sure exactly why it went wrong. Other people had experience with trying to measure success in the field, share experienceMay need to look at a certain kind of EU applications. Spreadsheets are mainly defined tasks, whereas tailoring is a different issue. These areas may not have comparable measures across instances of the case. Here you need to use other methods and types of measures. By emphasizing one side of the method question, you are only asking certain kinds of questionsOne of the major difficulties in introducing tool in field is that users get annoyed when it doesnt work. You cant get good results with a semi-finished prototype. Need to have almost product-quality prototype,Singer WEUSE IV,28,Important to use real spreadsheets with real errors. Powell, et al. made conclusion that you shouldnt look at cell errors, but rather cell error instances. EUSPRIG.com more practitioner focused than this group. Connection between SE and EUSE - strong connection on two levels - system that are to be developed by non-professionals need to modularized more. Very difficult to decide what is the right decomposition. In the future SE will need to watch what users and how requirements will evolve. Techniques are needed there. SE can perhaps benefit from the methodological practices of EUSE - where there is a huge degree of expertise in looking at practice. Understanding of how practices in software are evolved.,Singer WEUSE IV,29,Speculate about how conclusions would be different if gender wasnt a factor. Could not have found the same things if hadnt thought about gender. Was a basic emphasis in all work because of theoretical perspective. Professional and EUSE - Good case in point was curb cuts. Now we all benefit from curb cuts even though they were designed for disabled people. Some times when you look at a case that you havent looked at before, you can come back to the majority and