版权说明:本文档由用户提供并上传,收益归属内容提供方,若内容存在侵权,请进行举报或认领
文档简介
MeasuringthePerformanceofCodeProducedwithGitHubCopilot
by
DanielErhabor
Athesis
presentedtotheUniversityofWaterlooinfulfillmentofthe
thesisrequirementforthedegreeofMasterofMathematics
inComputerScience
Waterloo,Ontario,Canada,2022
©DanielErhabor2022
PAGE\*roman
viii
Author’sDeclaration
IherebydeclarethatIamthesoleauthorofthisthesis.Thisisatruecopyofthethesis,includinganyrequiredfinalrevisions,asacceptedbymyexaminers.
Iunderstandthatmythesismaybemadeelectronicallyavailabletothepublic.
Abstract
GitHubCopilotisanartificiallyintelligentprogrammingassistantusedbymanydevel-opers.WhileafewstudieshaveevaluatedthesecurityrisksofusingCopilot,therehasnotbeenanystudytoshowifitaidsdevelopersinproducingcodewithbetterperformance.WeevaluatetheperformanceofcodeproducedwhendevelopersuseGitHubCopilotver-suswhentheydonot.Tothisend,weconductedauserstudywith32participantswhereeachparticipantsolvedtwoC++programmingproblems,onewithCopilotandtheotherwithoutitandmeasuredtherunningtimeoftheparticipants’solutionsonourtestdata.OurresultssuggestthatusingCopilotcanproducecodewithasignificantlyslowerrunningtime.
Acknowledgements
Iwanttothankmysupervisors,MeiyappanNagappan,SamerAl-Kiswany,mycollab-oratoronthiswork,SreeharshaUdayashankar,theparticipantsforparticipatinginthestudy,membersofWASLandSWAGresearchgroups,andotherpeoplewhoadvisedonthingsrelatedtothiswork.
AsamemberoftheUniversityofWaterloo,IacknowledgethatthisworktookplaceonthetraditionalterritoryoftheNeutral,AnishinaabeandHaudenosauneepeoples.
Dedication
Idedicatethisthesistomyfriendsandfamily.
TableofContents
Author’sDeclaration
ii
Abstract
iii
Acknowledgements
iv
Dedication
v
ListofFigures
ix
ListofTables
x
Introduction
1
BackgroundandRelatedWork
3
ProgrammingProblemsSolvedbyParticipants
5
Problemselection
................................ 5
ProblemA
.................................... 6
ProblemB
.................................... 7
ModelSolutionstotheProblems
9
SolutionA
.................................... 9
Level0
.................................. 9
Level1
10
Level2
11
Level3
11
SolutionB
12
Level0
12
Level1
13
Participants
15
ParticipantRecruitment
15
DifficultiesRecruitingProfessionals
15
ParticipantSummary
16
ExperimentDesign
19
OrderofSolvingtheProblems
19
SessionIntroductionandTutorial
19
Tasks
20
Timing
21
AftertheProblem
21
BriefPost-sessionInterview
21
Evaluation
22
RQ0-DoesusingCopilotinfluenceprogramcorrectness?
22
RQ1-IstherearunningtimedifferenceincodewhenusingGitHubCopilot?
23
Approach
23
Results
24
Discussion
24
RQ2-DoCopilot’ssuggestionsswaydeveloperstoorfromcodewithfaster
runningtime?
25
Approach
25
StatementLevelOptimizations&Open-coding
26
VideoAnalysis
28
Results
28
Discussion
29
RQ3-DocharacteristicsofCopilotusersinfluencetherunningtimewhen
itisused?
36
Approach
36
Results
37
Discussion
37
Conclusion
40
Limitations
40
Takeaways
41
References
42
APPENDICES
46
AFullDescriptionoftheProblemsGiventoParticipants
47
ProblemA
47
ProblemB
52
Screeningsurvey
58
Tutorial
59
ProgrammingSurveys
61
FirstProgrammingSurvey
61
SecondProgrammingSurvey
62
ListofFigures
OverviewofMethodology
........................... 2
CopilotinAction
................................ 4
DistributionofParticipants’DeveloperExperiencefromScreeningSurvey
inAppendix
B
17
DistributionofParticipants’FamiliaritywithC++fromScreeningSurvey
inAppendix
B
18
PlotofProblemComprehensionforProblemAvsB.”x”isthemeanof
the6-PointLikert-ScaleinTable
7.4
andAppendix
D
38
ListofTables
6.1
FactorialMatrixofmodexproblem
....................
19
6.2
PossibleOrdersofmodexproblem
.....................
20
7.1
TableofInvalidRuns
..............................
23
7.2
TableofStatement-levelOptimizations&remarksforProblemA
.....
27
7.3
TableofStatement-levelOptimizations&remarksforProblemB
.....
28
7.4
TableofInputDatatoCorrelationMatrix
..................
36
7.5
CorrelationmatrixforproblemA
.......................
36
7.6
CorrelationmatrixforproblemB
.......................
37
B.1
ScreeningSurvey
................................
58
D.1
FirstProgrammingSurvey
...........................
62
D.2
SecondProgrammingSurveyPlusDemographics
...............
64
PAGE
10
Chapter1Introduction
Advancesinnaturallanguageprocessinganddeeplearninghaveresultedinlargelanguagemodels(LLMs)thatcangeneratecodefromfree-formtext.Withthis,languagemodelslikeGPT-3[
27
]havebeenextendedtowhatXuetal.[
35
]havetermedNatural-Language-to-Code(NL2Code)generators.Notably,OpenAI’sextensionoftheGPT-3languagemodel,Codex[
28
],andtheproduction-readyproductderivedfromit,GitHubCopilot[
1
],arepopularexamplesofNL2Codetoolsinusetoday.WhilesomestudieshaveshownthatdevelopersgenerallymayhaveapositiveexperienceusingGitHubCopilot,othershaveshownthatitcouldgeneratepotentiallyvulnerablecode.Wepresentthefirst-everevaluationofCopilotfromaperformanceperspectiveinsystemsprogramming.Wecon-ductedthefirstuserstudyonCopilottoevaluatetherunningtimeofthecodegeneratedwhendevelopersuseit.Withtheresultsfromourstudy,wehopetoanswerthefollowingresearchquestions:
RQ0:DoesusingCopilotinfluenceprogramcorrectness?
RQ1:IstherearunningtimedifferenceincodewhenusingGitHubCopilot??
RQ2:DoCopilot’ssuggestionsswaydeveloperstoorfromcodewithfasterrunningtime?
RQ3:DocharacteristicsofCopilotusersinfluencetherunningtimewhenitisused?
Toanswerourresearchquestions,weconductauserstudywith32participants,whereeachparticipantsolvedtwoprogrammingproblemsinC++,oneproblemwassolvedwith
Generatingtaskdesignfromproblems
A.cpp
B.cpp
runtime
surveydata
A
B.cpp
A.cpp
P1
B.cpp
A.cpp
P2P3
videoanalysis
P4
B
A.cpp
B.cpp
opencoding
RQ1
RQ3
RQ2
Problemselection
Assigningproblemstoparticipantstocounterbalanceorder-effects
Figure1.1:OverviewofMethodology
Copilotandtheotherwassolvedwithoutit.OurfindingsindicatethatusingCopilotresultedincodewithaslowerrunningtime.
Thethesisisorganizedinthefollowingway:WebrieflygooversomebackgroundrelatedtoGitHubCopilotandsomerelatedworkinChapter
2
.TheprocessofcreatingtheproblemstheparticipantswouldsolveandtherationalebehindchoosingtheproblemsisdescribedinChapter
3
.OurmodelsolutionstotheproblemsareelaboratedinChapter
4
givingcontexttotheproblems.AsummaryoftheparticipantrecruitmentprocessandtheparticipantsisdescribedinChapter
5
.WethenpresenttheexperimentdesignindetailinChapter
6
.AnoverviewofthemethodologycanbeseeninFigure
1.1
.Penultimately,weanalyzeanddiscusstheexperiment’sresults,answeringourresearchquestionsinChapter
7
.Finally,inChapter
8
,wetalkaboutthetakeawaysandlimitationsofourstudyandpotentialfuturedirections.
Chapter2
BackgroundandRelatedWork
GitHubCopilot,theproduction-readytoolbasedontheCodexmodelbyOpenAI,canbeusedasaVisualStudioCodeextensiontosuggestcodesnippetstouserswhentheextensionisactivated.Inthisway,userscanreceivesuggestionsbystartingtowritethecodeorbywritingcomments;eitherway,Copilotwillsuggestsomesnippets[
1
].SeeFigure
2.1
foranexampleofCopilotinaction.
OneoftheearlystudiesonCopilotwasbyPearceetal.[
31
],wheretheywantedtounderstandhowoftensuggestionsfromCopilotwerevulnerableandthecontextsthatmadeCopilotsuggestvulnerablecode.Toachievethis,theypromptedCopilottosuggestcodeinscenarioswheretheresultantsuggestionscouldhavebeenvulnerableormoresecure.Oftheprogramsproducedinresponsetothepotentiallyvulnerablescenarios,40%werevulnerable.
AstudybySandovaletal.[
32
]incollaborationwithPearcefrom[
31
]wantedtoassessthesecurityofcodewrittenbystudentprogrammerswhenassistedbyanNL2Codeassistant(OpenAI’scode-cushman-001model)likeCopilot.Theyconductedabetween-subjectsstudywith58computersciencestudentswhereparticipantsweretaskedwithimplementingoperationsofaSingly-LinkedListinC.Contraryto[
31
],theirresultsdidnotshowthatCopilothadaconclusiveimpactonsecurity.
Vaithilingametal.[
34
]conductedauserstudyon24participantstounderstandhowprogrammersperceiveanduseCopilot;theyfoundthatprogrammerspreferredtouseCopilotintheirday-to-dayprogrammingtasksandfoundithelpfulasastartingpoint.
Figure2.1:CopilotinAction
Chapter3
ProgrammingProblemsSolvedbyParticipants
FollowinginthesameveinasPearsonetal.[
31
],weprovided”incomplete”codeforpar-ticipantstoimplementasasolutiontoagivenproblem.By”incomplete”,wemeanthatweprovidedcodestubsandaccompanyingdocumentationforthestubsparticipantswereaskedtoimplementduringthestudy.Wecallthestubs”prompts”or”problems”andwilluseeitherofthosetermsinterchangeablythroughoutthiswork.ThesepromptswereprovidedtoparticipantsintheformofaCPPfilethatcontainedthefunctiondeclaration,theunimplementedfunctiondefinitionthatparticipantswereexpectedtoimplement,i.e.,theprimaryfunction,initializationfunctionsandsanitycheckstoverifycorrectness.Amainfunctionwasalsoprovidedasanentrypointtocalltheinitializationfunctions,theprimaryfunction,andthesanitychecksintheappropriateorder.
Problemselection
Wechosetwoproblemdomainsforourprogrammingproblems,file-systemoperationsandmultithreadedprogramming.Wechosethesetwoareasbecauseproblemsinthosedomainstendtohaveadirectimpactonapplicationperformance.WithfileI/Ooperationsaccountingforabout30%-80%ofinteractionsinnetworkedfilesystems[
29
],thereisaneedforfilesystemoperationstobefastonstoragedevices[
33
].Choosingaproblemrelatedtofilesystemsreflectsthisdemand.Additionally,sincemoderncomputingismovingtowardsamoreparalleldomain,thereisaneedtounderstandthebottlenecksofmultithreaded
applications[
30
]andoptimizeaccordingly.Toreflectthis,wechoseaproblemrelatedtofalsesharing,atypicalmulti-threadingoptimizationproblemthatisrelativelypopular[
2
].
Wechoseproblemsthatfitthefollowingcriteria:(1)theproblemmusthavemorethanonesolutionwhereeachsolutiondiffersnotincorrectnessbutperformance,(2)TheproblemshouldbesolvablewithorwithoutCopilotassistancein30minutes.ProblemAwasinthefile-systemoperationsarea,andproblemBwasinthemulti-threadingspace.
ProblemA
Forthisproblem,participantswereaskedtoreadmanyrecordsfromthree1GBlargetextfilesandwriteeachrecordtotheappropriatefilecombination.Afilecombinationisstructthatcontainedafileidentifier,abuffertowritetherecordto,andtheoffsetfortheassociatedfile.Arecordisasequenceof5000bytes.Forthisproblem,participantsreceivedaCPPfileforpromptAandthreelargetextfiles.WeprovideasummaryofrelevantdeclarationsformorecontexttotheprobleminListing
3.1
.ThefullfunctionsignaturesandtheentiretyoftheCPPfilewiththeaccompanyingdocumentationforpromptAgiventoparticipantsisinAppendix
A.1
.
1 #defineRECORD_SIZE5000
2 #defineNUM_RECORDS500000
3
4 conststd::vector<std::string>FILE_NAMES={
5 "large_file_1.txt","large_file_2.txt","large_file_3.txt"};
6
7 structFileCombo{
8 intfileId;
9 intoffset;
10 charbuffer[RECORD_SIZE+1];
11 };
12
13 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
14 //YOURCODEGOESHERE
15 }
Listing3.1:SummaryofProblemA
ProblemB
Forthisproblem,participantswereaskedtouseacertainamountofthreadstosetallthevaluesinasourcearraybuffertozerowhilesettingallthevaluesinadestinationarraybuffertoaparticularvalue.However,theywerenotallowedtouseassignmentoperations,i.e.,moveandcopysemanticswerenotallowedoneitherthesourcearraybufferorthedestinationarraybuffer.Participantswereonlyallowedtoincrementordecrementthevaluesintherespectivearraybufferstosolvethetask.Thisrestrictionwasinplacebecausewewantedthreadstorepeatedlywritetoaniteminthearrayandthusshowthefalsesharingeffect(dependingontheimplementation).WeprovideasummaryofrelevantdeclarationsformorecontexttotheprobleminListing
3.2
.ThefullfunctionsignaturesandtheentiretyoftheCPPfilewiththeaccompanyingdocumentationforpromptBgiventoparticipantsareinAppendix
A.2
.
1 constintINIT_SRC_VAL=(1<<17);
2 constintSIZE=(1<<11);
3 constintTHREAD_COUNT=4;
4
5 structItem{
6 private:
7 intval;
8 Item(constItem&);
9 Item(Item&&);
10 Item&operator=(constItem&);
11 Item&operator=(Item&&);
12 public:
13 Item(){val=0;}
14 Item(inti){val=i;}
15
16 intget(){returnval;}
17
18 voidoperator++(){++val;}
19 voidoperator++(int){val++;}
20
21 voidoperator--(){--val;}
22 voidoperator--(int){val--;}
23 };
24
25 Itemsrc[SIZE];
26 Itemdst[SIZE];
27
28 voidschedule(){
29 //YOURCODEGOESHERE
30 }
Listing3.2:SummaryofProblemB
Chapter4
ModelSolutionstotheProblems
Wecreatedwhatweterm”model”solutionstotheproblems.Becausetherewasmorethanonesolutiontoeachproblem,eachsolutionwederiveddifferedonlyinperformanceandnotcorrectness.WeitemizeoursolutionshereandcategorizethemintoLevel0(L0),Level1(L1),Level2(L2),andLevel3(L3)forproblemAandLevel0(L0)andLevel1(L1)forproblemB.
SolutionA
Level0
Weconsideranaiveimplementationwhereincallstoopen,seek,read,andclosearemadeforeachfileComboinfileCombos.(SeeListing
4.1
)
1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
2 for(auto&fileCombo:fileCombos){
3 ifstreamin;
4 in.open(FILE_NAMES[fileCombo.fileId],std::ios::binary);
5 in.seekg(fileCombo.offset);
6 in.read(fileCombo.buffer,RECORD_SIZE);
7 in.close();
8 }
9 }
Listing4.1:OurNaiveLevel0(L0)SolutiontoProblemA
Level1
Onestepfurtherfromthenaiveimplementationisacknowledgingthatonlythreefilesarebeinginteractedwith;thus,wedonotneedtoopenandcloseafileforeachfileComboinfileCombos.OuroptimizationinvolvesopeningallthefilesinFILE_NAMESfirst,thenprocessingeachfileComboinfileCombos,thenclosingallthefiles.(SeeListing
4.2
)
1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
2 std::vector<ifstream>files(FILE_NAMES.size());
3 for(inti=0;i<FILE_NAMES.size();++i){
4 files[i].open(FILE_NAMES[i],std::ios::binary);
5 }
6 for(FileCombo&fc:fileCombos){
7 files[fc.fileId].seekg(fc.offset);
8 files[fc.fileId].read(fc.buffer,RECORD_SIZE);
9 }
10 for(ifstream&f:files){
11 f.close();
12 }
13
14 }
Listing4.2:OurLevel1(L1)SolutiontoProblemA
Level2
AsafurtherstepfromL1,inthisimplementation,wesortthefileCombosbyfileIdandbreaktiesbyoffset.Thisway,readingtherecordfromanoffsetinaspecificfilewillbesequentialandnotrandom.(SeeListing
4.3
).
1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
2 std::sort(fileCombos.begin(),fileCombos.end(),
3 [](constFileCombo&a,constFileCombo&b){
4 if(a.fileId!=b.fileId){
5 returna.fileId<b.fileId;
6 }
7 returna.offset<b.offset;
8 });
9
10 for(FileCombo&fc:fileCombos){
11 ifstreamin;
12 in.open(FILE_NAMES[fileCombo.fileId],std::ios::binary);
13 in.seekg(fc.offset);
14 in.read(fc.buffer,RECORD_SIZE);
15 in.close();
16 }
17 }
Listing4.3:OurLevel2(L2)SolutiontoProblemA
Level3
AstepfurtherfromL2isacombinationoftheL1optimizationwedidin
4.1.2
andtheL2optimizationwedidin
4.1.3
.(SeeListing
4.4
).
1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
2 std::vector<ifstream>files(FILE_NAMES.size());
3 for(inti=0;i<FILE_NAMES.size();++i){
4 files[i].open(FILE_NAMES[i],std::ios::binary);
5 }
6
7 std::sort(fileCombos.begin(),fileCombos.end(),
8 [](constFileCombo&a,constFileCombo&b){
9 if(a.fileId!=b.fileId){
10 returna.fileId<b.fileId;
11 }
12 returna.offset<b.offset;
13 });
14
15 for(FileCombo&fc:fileCombos){
16 files[fc.fileId].seekg(fc.offset);
17 files[fc.fileId].read(fc.buffer,RECORD_SIZE);
18 }
19
20 for(ifstream&f:files){
21 f.close();
22 }
23
24 }
Listing4.4:OurLevel3(L3)SolutiontoProblemA
SolutionB
Level0
Weconsideranaiveimplementationtobeasolutionwhereeachthreadstartsatthere-spectiveindices0,1,2,and3(whereTHREAD_COUNTis4)inthesrcanddstarrays.EachthreadthendecrementsandincrementstheIteminsrcanddst,respectively.(SeeList-ing
4.5
).AfterprocessingtherespectiveItem,eachthreadmovesTHREAD_COUNTstepsuntilthenextindex,i.e.,4,5,6,and7andprocessestheItemtherein.Weconsiderthisthe
naivesolutionbecausefalsesharingispresentbecauseeachthreadinvalidatesthesame64-bytecachelinewhendecrementingandincrementingtheItematsrcanddstarrays.
1 voidwork(intstart){
2 for(inti=start;i<SIZE;i+=THREAD_COUNT){
3 for(intj=0;j<INIT_SRC_VAL;++j){
4 --src[i];
5 ++dst[i];
6 }
7 }
8 }
9
10 voidschedule(){
11 std::threadthreads[THREAD_COUNT];
12 for(inti=0;i<THREAD_COUNT;++i){
13 threads[i]=std::thread(work,i);
14 }
15 for(inti=0;i<THREAD_COUNT;++i){
16 std::threads[i].join();
17 }
18 }
Listing4.5:OurNaiveLevel0(L0)SolutiontoProblemB
Level1
Oursecondoptimizationlevelistoavoidfalsesharingbydividingeacharray(srcanddst)intoTHREAD_COUNTslicesandassigningasinglethreadtoprocesseachIteminthatslice.WhileweacknowledgethataligningtheItemstructdefinitioninListing
3.2
to64bytes(thecachelinesize)couldbeanotherwayofavoidingfalsesharing,wechosenottogiveparticipantstheflexibilityofmodifyingthestructdefinitionandthuspotentiallyviolatingthetimelimitconstraintfortheproblem.(SeeListing
4.6
)
1
voidwork(intstart,intend){
2
for(inti=start;i<end;++i){
3
for(intj=0;j<INIT_SRC_VAL;++j){
4
--src[i];
5
++dst[i];
6
}
7
}
8
}
9
10
voidschedule(){
11
intslice=SIZE/THREAD_COUNT;
12
std::threadthreads[THREAD_COUNT];
13
for(inti=0;i<THREAD_COUNT;++i){
14
threads[i]=std::thread(work,i*slice,(i+1)*slice);
15
}
16
17
for(inti=0;i<THREAD_COUNT;++i){
18
threads[i].join();
19
}
20
}
Listing4.6:OurLevel1(L1)SolutiontoProblemB
Chapter5Participants
ParticipantRecruitment
Participantswererecruitedmainlyviathemailinglistforcomputersciencegraduatestu-dentsandsnowballedtootherinterestedparticipants.Wefocusedonsystemsdevelopers.Weconsiderparticipantsassystemsdevelopersiftheyhadtakenasystemscourseinclud-ingbutnotlimitedtoOperatingSystems,DistributedSystems,orComputerNetworking.Wealsoconsideredindividualsassystemdevelopersiftheywereinvolvedinsystemsde-velopmentprofessionally,withopen-sourcecontributionsincluded.
Tobeeligibleforthestudy,potentialparticipantsneededaccesstoaninternetbrowserandGitHubCopilotonVSCodeatthetime.Theyalsomustbeasystemdeveloperasdescribedabove,musthavehadatleastafewmonthsofprogrammingexperience,andmusthavehadsomefamiliaritywiththeC++programminglanguage.Additionally,tobeeligible,participantscouldnotbeemployedbyOpenAIorGitHuborinvolvedwiththedevelopmentofGitHubCopilotatthetime.
Tocheckifpotentialparticipantswereeligibletoparticipate,theyweresentaQualtricsscreeningsurveyaftertheyhadreadandsignedtheconsentformdeclaringtheirintenttoparticipate.DetailsofthescreeningsurveycanbefoundinAppendix
B
.
DifficultiesRecruitingProfessionals
Atthehalfwaypointofourdesiredparticipantgoal,wepausedparticipantrecruitmenttoanalyzethepreliminarydatawehadobtained.Onlookingatthesnapshotofparticipants’
solutionstoproblemA,wenoticedthatnotasingleparticipanthadimplementedanyofthethreelevelsofoptimizationswehadconsideredwhendesigningtheproblem.Atthetime,mostoftheparticipantshadbeengraduatestudentswithsoundsystemsbackgrounds,i.e.,theywerepartofaresearchgroupthatfocusedonsystems.However,wedecidedtodiversifyourparticipantpoolbyincludingprofessionalsystemsdevelopers.Theinitialprocessofattemptingtorecruitprofessionalsystemsdevelopersstartedwithcontactingalumsoftheaffiliateduniversitywhowereknowntobeworkingassystemsdevelopers.Additionally,welookedforcontributorstosystemsprojectsonGitHubthatwereprimar-ilyimplementedinC++.Theadvancedsearchfeaturewasusedtofindprojectsthatcontainedthekeywords”systems”,”operatingsystems,”or”database”.Wefine-grainedoursearchtoprojectswithadedicatedsocialplatformwhereinterestedpartiesconnect,i.e.,Discord[
4
]andInternetRelayChat(IRC)[
8
].ProjectssuchasSerenityOS[
14
]andSkiftOS[
15
]hadactiveDiscordcommunities;however,therewasapaucityofinterestedpotentialparticipantsinthestudy.
AttemptstogarnerinterestinthestudyfromsaidprojectcontributorsweremetwitheitherbacklashorsuggestionstoreachouttootherDiscordcommunitiessuchastheosdev(OperatingSystemsDevelopment)[
12
]discordandtheassociatedIRC.UponinteractingwiththeosdevcommunityontheDiscordandIRCplatforms,therewasageneralunwill-ingnesstoparticipateinthestudy,withcommunitymemberscitingpotentialcopyrightissueswithCopilotandothernegativeperceptionsofGitHubCopilot,GitHub,andMi-crosoft.Thankfully,recruitmenteffortspaidoffasafew(lessthanwewouldhaveliked)professionalswerewillingtoparticipateinourstudyandthusmetourdesiredparticipantgoal.
ParticipantSummary
Werecruitedatotalof32participantsforthestudy,where8wereprofessionalsinsystemsprogrammingorcontributorstoopen-sourcesystemsprojects.23weregraduatestudentswithasystemsresearchareaatthetimeofparticipatingandonewasasessionallecturerbutwaspreviouslyagraduatestudentwithasystemsresearchfocus.Thedistributionoftheparticipants’experienceisinFigure
5.1
andtheirfamiliaritywithC++isinFigure
5.2
.Participantswerecompensated$50fortheirtimeandthestudywasapprovedbyResearchEthicsBoard(REB#44162)attheaffiliateduniversity.
13
11
9
count
7
5
3
1
3.0 3.5 4.0 4.5 5.0
devExp
Figure5.1:DistributionofParticipants’DeveloperExperiencefromScreeningSurveyinAppendix
B
17
15
13
11
count
9
7
5
3
1
2 3 4 5
familiarCPP
Figure5.2:DistributionofParticipants’FamiliaritywithC++fromScreeningSurveyinAppendix
B
Chapter6ExperimentDesign
OrderofSolvingtheProblems
Givenourwithin-subjectsexperimentaldesignwhereoneparticipantsolvesoneproblemwithCopilotandthentheotherproblemwithoutit,weneededtoensurethatanyordereffectsarecounterbalancedacrossall32participants.Tothisend,wepresentafactorialmatrixwheretheprompts(AandB)arethecolumnlabels,andthemodes(CandNC)aretherowlabelswhichindicateusingCopilotandnotusingCopilotrespectively.(SeeTable
6.1
).Wefurtherexpandthismatrixtotheproductofpromptsandmodes.Fourpossibleordersofproblemxmodearegenerated(SeeTable
6.2
).
TheordersinTable
6.2
enforcedarequirementthatourparticipantpoolbeamultipleoffour.Hence,werecruited32participantsforthestudy.
axes
A
B
C
NC
CxA
NCxA
CxB
NCxB
Table6.1:FactorialMatrixofmodexproblem
SessionIntroductionandTutorial
Thesessionwasdoneremotelyonanonlineconferencingplatform.Itstartedwiththefacilitatorintroducingthestudy,confirmingtheparticipant’sconsenttoparticipate,and
#
first
second
ParticipantID
1
CxA
NCxB
P1
2
NCxB
CxA
P2
3
NCxA
CxB
P3
4
CxB
NCxA
P4
Table6.2:PossibleOrdersofmodexproblem
thenconfirmingtheparticipant’snumber(theparticipantIDgiventotheparticipantonceeligibilityandconsentweregivenbeforethesession).Thefacilitatorthencontinuedbyexplainingtheoverviewofparticipantresponsibility.Theythenrequestedtheparticipant’sconsenttorecordtheaudioandthescreenduringthesession.Finally,theexperimentergaveatutorialonwhatwasexpected,fromopeningtheprobleminVSCodetousingCopilottoaccept,reject,andviewallsuggestionsandzippingtheeditedcodefiles.(SeeAppendix
C
formoredetailsonthetutorial)
Tasks
ParticipantsweregiventwoC++programmingproblemstosolvewithin30minuteseach.Eachpromptwasself-containedwithinaCPPfilewithinacompressedfolder,
温馨提示
- 1. 本站所有资源如无特殊说明,都需要本地电脑安装OFFICE2007和PDF阅读器。图纸软件为CAD,CAXA,PROE,UG,SolidWorks等.压缩文件请下载最新的WinRAR软件解压。
- 2. 本站的文档不包含任何第三方提供的附件图纸等,如果需要附件,请联系上传者。文件的所有权益归上传用户所有。
- 3. 本站RAR压缩包中若带图纸,网页内容里面会有图纸预览,若没有图纸预览就没有图纸。
- 4. 未经权益所有人同意不得将文件中的内容挪作商业或盈利用途。
- 5. 人人文库网仅提供信息存储空间,仅对用户上传内容的表现方式做保护处理,对用户上传分享的文档内容本身不做任何修改或编辑,并不能对任何下载内容负责。
- 6. 下载文件中如有侵权或不适当内容,请与我们联系,我们立即纠正。
- 7. 本站不保证下载资源的准确性、安全性和完整性, 同时也不承担用户因使用这些下载资源对自己和他人造成任何形式的伤害或损失。
最新文档
- 上海市嘉定区2023-2024学年中考三模数学试题含解析
- 陕西省榆林市定边县2023-2024学年毕业升学考试模拟卷数学卷含解析
- 山西省左玉县2024届中考数学押题卷含解析
- 2024-2030年中国壁纸行业消费趋势及营销趋势预测报告
- 2024-2030年中国墙体保温材料行业市场发展前景预测及投资战略决策报告
- 2024-2030年中国填料塔尾气吸收装置项目可行性研究报告
- 2024-2030年中国培加帕酶项目可行性研究报告
- 2024-2030年中国地下停车场行业市场发展现状及发展趋势与投资前景研究报告
- 2024-2030年中国国内旅行行业发展分析及发展前景与趋势预测研究报告
- 2024年4月自考00906电子商务网站设计原理练习考题含解析
- 中班绘本ppt彩虹色花
- 刮腻子涂料顶棚施工方案
- 小学五年级下册科学课件-1.1《水的变化》3人教版(21张)ppt课件
- 如何正确使用吸入剂和注意事项ppt
- 小学美术国测(国家义务教育质量监测)复习内容
- 四年级数学教师家长会PPT课件
- DN800顶管顶进阻力计算和后背墙设计
- 矩阵可对角化的判定条件及推广
- 北京中考数学一模——海淀答题卡
- 商业综合体、购物中心、百货商场商业运营项目收益测算模板
- 承插型盘扣式钢管脚手架验收表
评论
0/150
提交评论