版权说明:本文档由用户提供并上传,收益归属内容提供方,若内容存在侵权,请进行举报或认领
文档简介
MeasuringthePerformanceofCodeProducedwithGitHubCopilot
by
DanielErhabor
Athesis
presentedtotheUniversityofWaterlooinfulfillmentofthe
thesisrequirementforthedegreeofMasterofMathematics
inComputerScience
Waterloo,Ontario,Canada,2022
©DanielErhabor2022
PAGE\*roman
viii
Author’sDeclaration
IherebydeclarethatIamthesoleauthorofthisthesis.Thisisatruecopyofthethesis,includinganyrequiredfinalrevisions,asacceptedbymyexaminers.
Iunderstandthatmythesismaybemadeelectronicallyavailabletothepublic.
Abstract
GitHubCopilotisanartificiallyintelligentprogrammingassistantusedbymanydevel-opers.WhileafewstudieshaveevaluatedthesecurityrisksofusingCopilot,therehasnotbeenanystudytoshowifitaidsdevelopersinproducingcodewithbetterperformance.WeevaluatetheperformanceofcodeproducedwhendevelopersuseGitHubCopilotver-suswhentheydonot.Tothisend,weconductedauserstudywith32participantswhereeachparticipantsolvedtwoC++programmingproblems,onewithCopilotandtheotherwithoutitandmeasuredtherunningtimeoftheparticipants’solutionsonourtestdata.OurresultssuggestthatusingCopilotcanproducecodewithasignificantlyslowerrunningtime.
Acknowledgements
Iwanttothankmysupervisors,MeiyappanNagappan,SamerAl-Kiswany,mycollab-oratoronthiswork,SreeharshaUdayashankar,theparticipantsforparticipatinginthestudy,membersofWASLandSWAGresearchgroups,andotherpeoplewhoadvisedonthingsrelatedtothiswork.
AsamemberoftheUniversityofWaterloo,IacknowledgethatthisworktookplaceonthetraditionalterritoryoftheNeutral,AnishinaabeandHaudenosauneepeoples.
Dedication
Idedicatethisthesistomyfriendsandfamily.
TableofContents
Author’sDeclaration
ii
Abstract
iii
Acknowledgements
iv
Dedication
v
ListofFigures
ix
ListofTables
x
Introduction
1
BackgroundandRelatedWork
3
ProgrammingProblemsSolvedbyParticipants
5
Problemselection
................................ 5
ProblemA
.................................... 6
ProblemB
.................................... 7
ModelSolutionstotheProblems
9
SolutionA
.................................... 9
Level0
.................................. 9
Level1
10
Level2
11
Level3
11
SolutionB
12
Level0
12
Level1
13
Participants
15
ParticipantRecruitment
15
DifficultiesRecruitingProfessionals
15
ParticipantSummary
16
ExperimentDesign
19
OrderofSolvingtheProblems
19
SessionIntroductionandTutorial
19
Tasks
20
Timing
21
AftertheProblem
21
BriefPost-sessionInterview
21
Evaluation
22
RQ0-DoesusingCopilotinfluenceprogramcorrectness?
22
RQ1-IstherearunningtimedifferenceincodewhenusingGitHubCopilot?
23
Approach
23
Results
24
Discussion
24
RQ2-DoCopilot’ssuggestionsswaydeveloperstoorfromcodewithfaster
runningtime?
25
Approach
25
StatementLevelOptimizations&Open-coding
26
VideoAnalysis
28
Results
28
Discussion
29
RQ3-DocharacteristicsofCopilotusersinfluencetherunningtimewhen
itisused?
36
Approach
36
Results
37
Discussion
37
Conclusion
40
Limitations
40
Takeaways
41
References
42
APPENDICES
46
AFullDescriptionoftheProblemsGiventoParticipants
47
ProblemA
47
ProblemB
52
Screeningsurvey
58
Tutorial
59
ProgrammingSurveys
61
FirstProgrammingSurvey
61
SecondProgrammingSurvey
62
ListofFigures
OverviewofMethodology
........................... 2
CopilotinAction
................................ 4
DistributionofParticipants’DeveloperExperiencefromScreeningSurvey
inAppendix
B
17
DistributionofParticipants’FamiliaritywithC++fromScreeningSurvey
inAppendix
B
18
PlotofProblemComprehensionforProblemAvsB.”x”isthemeanof
the6-PointLikert-ScaleinTable
7.4
andAppendix
D
38
ListofTables
6.1
FactorialMatrixofmodexproblem
....................
19
6.2
PossibleOrdersofmodexproblem
.....................
20
7.1
TableofInvalidRuns
..............................
23
7.2
TableofStatement-levelOptimizations&remarksforProblemA
.....
27
7.3
TableofStatement-levelOptimizations&remarksforProblemB
.....
28
7.4
TableofInputDatatoCorrelationMatrix
..................
36
7.5
CorrelationmatrixforproblemA
.......................
36
7.6
CorrelationmatrixforproblemB
.......................
37
B.1
ScreeningSurvey
................................
58
D.1
FirstProgrammingSurvey
...........................
62
D.2
SecondProgrammingSurveyPlusDemographics
...............
64
PAGE
10
Chapter1Introduction
Advancesinnaturallanguageprocessinganddeeplearninghaveresultedinlargelanguagemodels(LLMs)thatcangeneratecodefromfree-formtext.Withthis,languagemodelslikeGPT-3[
27
]havebeenextendedtowhatXuetal.[
35
]havetermedNatural-Language-to-Code(NL2Code)generators.Notably,OpenAI’sextensionoftheGPT-3languagemodel,Codex[
28
],andtheproduction-readyproductderivedfromit,GitHubCopilot[
1
],arepopularexamplesofNL2Codetoolsinusetoday.WhilesomestudieshaveshownthatdevelopersgenerallymayhaveapositiveexperienceusingGitHubCopilot,othershaveshownthatitcouldgeneratepotentiallyvulnerablecode.Wepresentthefirst-everevaluationofCopilotfromaperformanceperspectiveinsystemsprogramming.Wecon-ductedthefirstuserstudyonCopilottoevaluatetherunningtimeofthecodegeneratedwhendevelopersuseit.Withtheresultsfromourstudy,wehopetoanswerthefollowingresearchquestions:
RQ0:DoesusingCopilotinfluenceprogramcorrectness?
RQ1:IstherearunningtimedifferenceincodewhenusingGitHubCopilot??
RQ2:DoCopilot’ssuggestionsswaydeveloperstoorfromcodewithfasterrunningtime?
RQ3:DocharacteristicsofCopilotusersinfluencetherunningtimewhenitisused?
Toanswerourresearchquestions,weconductauserstudywith32participants,whereeachparticipantsolvedtwoprogrammingproblemsinC++,oneproblemwassolvedwith
Generatingtaskdesignfromproblems
A.cpp
B.cpp
runtime
surveydata
A
B.cpp
A.cpp
P1
B.cpp
A.cpp
P2P3
videoanalysis
P4
B
A.cpp
B.cpp
opencoding
RQ1
RQ3
RQ2
Problemselection
Assigningproblemstoparticipantstocounterbalanceorder-effects
Figure1.1:OverviewofMethodology
Copilotandtheotherwassolvedwithoutit.OurfindingsindicatethatusingCopilotresultedincodewithaslowerrunningtime.
Thethesisisorganizedinthefollowingway:WebrieflygooversomebackgroundrelatedtoGitHubCopilotandsomerelatedworkinChapter
2
.TheprocessofcreatingtheproblemstheparticipantswouldsolveandtherationalebehindchoosingtheproblemsisdescribedinChapter
3
.OurmodelsolutionstotheproblemsareelaboratedinChapter
4
givingcontexttotheproblems.AsummaryoftheparticipantrecruitmentprocessandtheparticipantsisdescribedinChapter
5
.WethenpresenttheexperimentdesignindetailinChapter
6
.AnoverviewofthemethodologycanbeseeninFigure
1.1
.Penultimately,weanalyzeanddiscusstheexperiment’sresults,answeringourresearchquestionsinChapter
7
.Finally,inChapter
8
,wetalkaboutthetakeawaysandlimitationsofourstudyandpotentialfuturedirections.
Chapter2
BackgroundandRelatedWork
GitHubCopilot,theproduction-readytoolbasedontheCodexmodelbyOpenAI,canbeusedasaVisualStudioCodeextensiontosuggestcodesnippetstouserswhentheextensionisactivated.Inthisway,userscanreceivesuggestionsbystartingtowritethecodeorbywritingcomments;eitherway,Copilotwillsuggestsomesnippets[
1
].SeeFigure
2.1
foranexampleofCopilotinaction.
OneoftheearlystudiesonCopilotwasbyPearceetal.[
31
],wheretheywantedtounderstandhowoftensuggestionsfromCopilotwerevulnerableandthecontextsthatmadeCopilotsuggestvulnerablecode.Toachievethis,theypromptedCopilottosuggestcodeinscenarioswheretheresultantsuggestionscouldhavebeenvulnerableormoresecure.Oftheprogramsproducedinresponsetothepotentiallyvulnerablescenarios,40%werevulnerable.
AstudybySandovaletal.[
32
]incollaborationwithPearcefrom[
31
]wantedtoassessthesecurityofcodewrittenbystudentprogrammerswhenassistedbyanNL2Codeassistant(OpenAI’scode-cushman-001model)likeCopilot.Theyconductedabetween-subjectsstudywith58computersciencestudentswhereparticipantsweretaskedwithimplementingoperationsofaSingly-LinkedListinC.Contraryto[
31
],theirresultsdidnotshowthatCopilothadaconclusiveimpactonsecurity.
Vaithilingametal.[
34
]conductedauserstudyon24participantstounderstandhowprogrammersperceiveanduseCopilot;theyfoundthatprogrammerspreferredtouseCopilotintheirday-to-dayprogrammingtasksandfoundithelpfulasastartingpoint.
Figure2.1:CopilotinAction
Chapter3
ProgrammingProblemsSolvedbyParticipants
FollowinginthesameveinasPearsonetal.[
31
],weprovided”incomplete”codeforpar-ticipantstoimplementasasolutiontoagivenproblem.By”incomplete”,wemeanthatweprovidedcodestubsandaccompanyingdocumentationforthestubsparticipantswereaskedtoimplementduringthestudy.Wecallthestubs”prompts”or”problems”andwilluseeitherofthosetermsinterchangeablythroughoutthiswork.ThesepromptswereprovidedtoparticipantsintheformofaCPPfilethatcontainedthefunctiondeclaration,theunimplementedfunctiondefinitionthatparticipantswereexpectedtoimplement,i.e.,theprimaryfunction,initializationfunctionsandsanitycheckstoverifycorrectness.Amainfunctionwasalsoprovidedasanentrypointtocalltheinitializationfunctions,theprimaryfunction,andthesanitychecksintheappropriateorder.
Problemselection
Wechosetwoproblemdomainsforourprogrammingproblems,file-systemoperationsandmultithreadedprogramming.Wechosethesetwoareasbecauseproblemsinthosedomainstendtohaveadirectimpactonapplicationperformance.WithfileI/Ooperationsaccountingforabout30%-80%ofinteractionsinnetworkedfilesystems[
29
],thereisaneedforfilesystemoperationstobefastonstoragedevices[
33
].Choosingaproblemrelatedtofilesystemsreflectsthisdemand.Additionally,sincemoderncomputingismovingtowardsamoreparalleldomain,thereisaneedtounderstandthebottlenecksofmultithreaded
applications[
30
]andoptimizeaccordingly.Toreflectthis,wechoseaproblemrelatedtofalsesharing,atypicalmulti-threadingoptimizationproblemthatisrelativelypopular[
2
].
Wechoseproblemsthatfitthefollowingcriteria:(1)theproblemmusthavemorethanonesolutionwhereeachsolutiondiffersnotincorrectnessbutperformance,(2)TheproblemshouldbesolvablewithorwithoutCopilotassistancein30minutes.ProblemAwasinthefile-systemoperationsarea,andproblemBwasinthemulti-threadingspace.
ProblemA
Forthisproblem,participantswereaskedtoreadmanyrecordsfromthree1GBlargetextfilesandwriteeachrecordtotheappropriatefilecombination.Afilecombinationisstructthatcontainedafileidentifier,abuffertowritetherecordto,andtheoffsetfortheassociatedfile.Arecordisasequenceof5000bytes.Forthisproblem,participantsreceivedaCPPfileforpromptAandthreelargetextfiles.WeprovideasummaryofrelevantdeclarationsformorecontexttotheprobleminListing
3.1
.ThefullfunctionsignaturesandtheentiretyoftheCPPfilewiththeaccompanyingdocumentationforpromptAgiventoparticipantsisinAppendix
A.1
.
1 #defineRECORD_SIZE5000
2 #defineNUM_RECORDS500000
3
4 conststd::vector<std::string>FILE_NAMES={
5 "large_file_1.txt","large_file_2.txt","large_file_3.txt"};
6
7 structFileCombo{
8 intfileId;
9 intoffset;
10 charbuffer[RECORD_SIZE+1];
11 };
12
13 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
14 //YOURCODEGOESHERE
15 }
Listing3.1:SummaryofProblemA
ProblemB
Forthisproblem,participantswereaskedtouseacertainamountofthreadstosetallthevaluesinasourcearraybuffertozerowhilesettingallthevaluesinadestinationarraybuffertoaparticularvalue.However,theywerenotallowedtouseassignmentoperations,i.e.,moveandcopysemanticswerenotallowedoneitherthesourcearraybufferorthedestinationarraybuffer.Participantswereonlyallowedtoincrementordecrementthevaluesintherespectivearraybufferstosolvethetask.Thisrestrictionwasinplacebecausewewantedthreadstorepeatedlywritetoaniteminthearrayandthusshowthefalsesharingeffect(dependingontheimplementation).WeprovideasummaryofrelevantdeclarationsformorecontexttotheprobleminListing
3.2
.ThefullfunctionsignaturesandtheentiretyoftheCPPfilewiththeaccompanyingdocumentationforpromptBgiventoparticipantsareinAppendix
A.2
.
1 constintINIT_SRC_VAL=(1<<17);
2 constintSIZE=(1<<11);
3 constintTHREAD_COUNT=4;
4
5 structItem{
6 private:
7 intval;
8 Item(constItem&);
9 Item(Item&&);
10 Item&operator=(constItem&);
11 Item&operator=(Item&&);
12 public:
13 Item(){val=0;}
14 Item(inti){val=i;}
15
16 intget(){returnval;}
17
18 voidoperator++(){++val;}
19 voidoperator++(int){val++;}
20
21 voidoperator--(){--val;}
22 voidoperator--(int){val--;}
23 };
24
25 Itemsrc[SIZE];
26 Itemdst[SIZE];
27
28 voidschedule(){
29 //YOURCODEGOESHERE
30 }
Listing3.2:SummaryofProblemB
Chapter4
ModelSolutionstotheProblems
Wecreatedwhatweterm”model”solutionstotheproblems.Becausetherewasmorethanonesolutiontoeachproblem,eachsolutionwederiveddifferedonlyinperformanceandnotcorrectness.WeitemizeoursolutionshereandcategorizethemintoLevel0(L0),Level1(L1),Level2(L2),andLevel3(L3)forproblemAandLevel0(L0)andLevel1(L1)forproblemB.
SolutionA
Level0
Weconsideranaiveimplementationwhereincallstoopen,seek,read,andclosearemadeforeachfileComboinfileCombos.(SeeListing
4.1
)
1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
2 for(auto&fileCombo:fileCombos){
3 ifstreamin;
4 in.open(FILE_NAMES[fileCombo.fileId],std::ios::binary);
5 in.seekg(fileCombo.offset);
6 in.read(fileCombo.buffer,RECORD_SIZE);
7 in.close();
8 }
9 }
Listing4.1:OurNaiveLevel0(L0)SolutiontoProblemA
Level1
Onestepfurtherfromthenaiveimplementationisacknowledgingthatonlythreefilesarebeinginteractedwith;thus,wedonotneedtoopenandcloseafileforeachfileComboinfileCombos.OuroptimizationinvolvesopeningallthefilesinFILE_NAMESfirst,thenprocessingeachfileComboinfileCombos,thenclosingallthefiles.(SeeListing
4.2
)
1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
2 std::vector<ifstream>files(FILE_NAMES.size());
3 for(inti=0;i<FILE_NAMES.size();++i){
4 files[i].open(FILE_NAMES[i],std::ios::binary);
5 }
6 for(FileCombo&fc:fileCombos){
7 files[fc.fileId].seekg(fc.offset);
8 files[fc.fileId].read(fc.buffer,RECORD_SIZE);
9 }
10 for(ifstream&f:files){
11 f.close();
12 }
13
14 }
Listing4.2:OurLevel1(L1)SolutiontoProblemA
Level2
AsafurtherstepfromL1,inthisimplementation,wesortthefileCombosbyfileIdandbreaktiesbyoffset.Thisway,readingtherecordfromanoffsetinaspecificfilewillbesequentialandnotrandom.(SeeListing
4.3
).
1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
2 std::sort(fileCombos.begin(),fileCombos.end(),
3 [](constFileCombo&a,constFileCombo&b){
4 if(a.fileId!=b.fileId){
5 returna.fileId<b.fileId;
6 }
7 returna.offset<b.offset;
8 });
9
10 for(FileCombo&fc:fileCombos){
11 ifstreamin;
12 in.open(FILE_NAMES[fileCombo.fileId],std::ios::binary);
13 in.seekg(fc.offset);
14 in.read(fc.buffer,RECORD_SIZE);
15 in.close();
16 }
17 }
Listing4.3:OurLevel2(L2)SolutiontoProblemA
Level3
AstepfurtherfromL2isacombinationoftheL1optimizationwedidin
4.1.2
andtheL2optimizationwedidin
4.1.3
.(SeeListing
4.4
).
1 voidreadFileCombos(std::vector<FileCombo>&fileCombos){
2 std::vector<ifstream>files(FILE_NAMES.size());
3 for(inti=0;i<FILE_NAMES.size();++i){
4 files[i].open(FILE_NAMES[i],std::ios::binary);
5 }
6
7 std::sort(fileCombos.begin(),fileCombos.end(),
8 [](constFileCombo&a,constFileCombo&b){
9 if(a.fileId!=b.fileId){
10 returna.fileId<b.fileId;
11 }
12 returna.offset<b.offset;
13 });
14
15 for(FileCombo&fc:fileCombos){
16 files[fc.fileId].seekg(fc.offset);
17 files[fc.fileId].read(fc.buffer,RECORD_SIZE);
18 }
19
20 for(ifstream&f:files){
21 f.close();
22 }
23
24 }
Listing4.4:OurLevel3(L3)SolutiontoProblemA
SolutionB
Level0
Weconsideranaiveimplementationtobeasolutionwhereeachthreadstartsatthere-spectiveindices0,1,2,and3(whereTHREAD_COUNTis4)inthesrcanddstarrays.EachthreadthendecrementsandincrementstheIteminsrcanddst,respectively.(SeeList-ing
4.5
).AfterprocessingtherespectiveItem,eachthreadmovesTHREAD_COUNTstepsuntilthenextindex,i.e.,4,5,6,and7andprocessestheItemtherein.Weconsiderthisthe
naivesolutionbecausefalsesharingispresentbecauseeachthreadinvalidatesthesame64-bytecachelinewhendecrementingandincrementingtheItematsrcanddstarrays.
1 voidwork(intstart){
2 for(inti=start;i<SIZE;i+=THREAD_COUNT){
3 for(intj=0;j<INIT_SRC_VAL;++j){
4 --src[i];
5 ++dst[i];
6 }
7 }
8 }
9
10 voidschedule(){
11 std::threadthreads[THREAD_COUNT];
12 for(inti=0;i<THREAD_COUNT;++i){
13 threads[i]=std::thread(work,i);
14 }
15 for(inti=0;i<THREAD_COUNT;++i){
16 std::threads[i].join();
17 }
18 }
Listing4.5:OurNaiveLevel0(L0)SolutiontoProblemB
Level1
Oursecondoptimizationlevelistoavoidfalsesharingbydividingeacharray(srcanddst)intoTHREAD_COUNTslicesandassigningasinglethreadtoprocesseachIteminthatslice.WhileweacknowledgethataligningtheItemstructdefinitioninListing
3.2
to64bytes(thecachelinesize)couldbeanotherwayofavoidingfalsesharing,wechosenottogiveparticipantstheflexibilityofmodifyingthestructdefinitionandthuspotentiallyviolatingthetimelimitconstraintfortheproblem.(SeeListing
4.6
)
1
voidwork(intstart,intend){
2
for(inti=start;i<end;++i){
3
for(intj=0;j<INIT_SRC_VAL;++j){
4
--src[i];
5
++dst[i];
6
}
7
}
8
}
9
10
voidschedule(){
11
intslice=SIZE/THREAD_COUNT;
12
std::threadthreads[THREAD_COUNT];
13
for(inti=0;i<THREAD_COUNT;++i){
14
threads[i]=std::thread(work,i*slice,(i+1)*slice);
15
}
16
17
for(inti=0;i<THREAD_COUNT;++i){
18
threads[i].join();
19
}
20
}
Listing4.6:OurLevel1(L1)SolutiontoProblemB
Chapter5Participants
ParticipantRecruitment
Participantswererecruitedmainlyviathemailinglistforcomputersciencegraduatestu-dentsandsnowballedtootherinterestedparticipants.Wefocusedonsystemsdevelopers.Weconsiderparticipantsassystemsdevelopersiftheyhadtakenasystemscourseinclud-ingbutnotlimitedtoOperatingSystems,DistributedSystems,orComputerNetworking.Wealsoconsideredindividualsassystemdevelopersiftheywereinvolvedinsystemsde-velopmentprofessionally,withopen-sourcecontributionsincluded.
Tobeeligibleforthestudy,potentialparticipantsneededaccesstoaninternetbrowserandGitHubCopilotonVSCodeatthetime.Theyalsomustbeasystemdeveloperasdescribedabove,musthavehadatleastafewmonthsofprogrammingexperience,andmusthavehadsomefamiliaritywiththeC++programminglanguage.Additionally,tobeeligible,participantscouldnotbeemployedbyOpenAIorGitHuborinvolvedwiththedevelopmentofGitHubCopilotatthetime.
Tocheckifpotentialparticipantswereeligibletoparticipate,theyweresentaQualtricsscreeningsurveyaftertheyhadreadandsignedtheconsentformdeclaringtheirintenttoparticipate.DetailsofthescreeningsurveycanbefoundinAppendix
B
.
DifficultiesRecruitingProfessionals
Atthehalfwaypointofourdesiredparticipantgoal,wepausedparticipantrecruitmenttoanalyzethepreliminarydatawehadobtained.Onlookingatthesnapshotofparticipants’
solutionstoproblemA,wenoticedthatnotasingleparticipanthadimplementedanyofthethreelevelsofoptimizationswehadconsideredwhendesigningtheproblem.Atthetime,mostoftheparticipantshadbeengraduatestudentswithsoundsystemsbackgrounds,i.e.,theywerepartofaresearchgroupthatfocusedonsystems.However,wedecidedtodiversifyourparticipantpoolbyincludingprofessionalsystemsdevelopers.Theinitialprocessofattemptingtorecruitprofessionalsystemsdevelopersstartedwithcontactingalumsoftheaffiliateduniversitywhowereknowntobeworkingassystemsdevelopers.Additionally,welookedforcontributorstosystemsprojectsonGitHubthatwereprimar-ilyimplementedinC++.Theadvancedsearchfeaturewasusedtofindprojectsthatcontainedthekeywords”systems”,”operatingsystems,”or”database”.Wefine-grainedoursearchtoprojectswithadedicatedsocialplatformwhereinterestedpartiesconnect,i.e.,Discord[
4
]andInternetRelayChat(IRC)[
8
].ProjectssuchasSerenityOS[
14
]andSkiftOS[
15
]hadactiveDiscordcommunities;however,therewasapaucityofinterestedpotentialparticipantsinthestudy.
AttemptstogarnerinterestinthestudyfromsaidprojectcontributorsweremetwitheitherbacklashorsuggestionstoreachouttootherDiscordcommunitiessuchastheosdev(OperatingSystemsDevelopment)[
12
]discordandtheassociatedIRC.UponinteractingwiththeosdevcommunityontheDiscordandIRCplatforms,therewasageneralunwill-ingnesstoparticipateinthestudy,withcommunitymemberscitingpotentialcopyrightissueswithCopilotandothernegativeperceptionsofGitHubCopilot,GitHub,andMi-crosoft.Thankfully,recruitmenteffortspaidoffasafew(lessthanwewouldhaveliked)professionalswerewillingtoparticipateinourstudyandthusmetourdesiredparticipantgoal.
ParticipantSummary
Werecruitedatotalof32participantsforthestudy,where8wereprofessionalsinsystemsprogrammingorcontributorstoopen-sourcesystemsprojects.23weregraduatestudentswithasystemsresearchareaatthetimeofparticipatingandonewasasessionallecturerbutwaspreviouslyagraduatestudentwithasystemsresearchfocus.Thedistributionoftheparticipants’experienceisinFigure
5.1
andtheirfamiliaritywithC++isinFigure
5.2
.Participantswerecompensated$50fortheirtimeandthestudywasapprovedbyResearchEthicsBoard(REB#44162)attheaffiliateduniversity.
13
11
9
count
7
5
3
1
3.0 3.5 4.0 4.5 5.0
devExp
Figure5.1:DistributionofParticipants’DeveloperExperiencefromScreeningSurveyinAppendix
B
17
15
13
11
count
9
7
5
3
1
2 3 4 5
familiarCPP
Figure5.2:DistributionofParticipants’FamiliaritywithC++fromScreeningSurveyinAppendix
B
Chapter6ExperimentDesign
OrderofSolvingtheProblems
Givenourwithin-subjectsexperimentaldesignwhereoneparticipantsolvesoneproblemwithCopilotandthentheotherproblemwithoutit,weneededtoensurethatanyordereffectsarecounterbalancedacrossall32participants.Tothisend,wepresentafactorialmatrixwheretheprompts(AandB)arethecolumnlabels,andthemodes(CandNC)aretherowlabelswhichindicateusingCopilotandnotusingCopilotrespectively.(SeeTable
6.1
).Wefurtherexpandthismatrixtotheproductofpromptsandmodes.Fourpossibleordersofproblemxmodearegenerated(SeeTable
6.2
).
TheordersinTable
6.2
enforcedarequirementthatourparticipantpoolbeamultipleoffour.Hence,werecruited32participantsforthestudy.
axes
A
B
C
NC
CxA
NCxA
CxB
NCxB
Table6.1:FactorialMatrixofmodexproblem
SessionIntroductionandTutorial
Thesessionwasdoneremotelyonanonlineconferencingplatform.Itstartedwiththefacilitatorintroducingthestudy,confirmingtheparticipant’sconsenttoparticipate,and
#
first
second
ParticipantID
1
CxA
NCxB
P1
2
NCxB
CxA
P2
3
NCxA
CxB
P3
4
CxB
NCxA
P4
Table6.2:PossibleOrdersofmodexproblem
thenconfirmingtheparticipant’snumber(theparticipantIDgiventotheparticipantonceeligibilityandconsentweregivenbeforethesession).Thefacilitatorthencontinuedbyexplainingtheoverviewofparticipantresponsibility.Theythenrequestedtheparticipant’sconsenttorecordtheaudioandthescreenduringthesession.Finally,theexperimentergaveatutorialonwhatwasexpected,fromopeningtheprobleminVSCodetousingCopilottoaccept,reject,andviewallsuggestionsandzippingtheeditedcodefiles.(SeeAppendix
C
formoredetailsonthetutorial)
Tasks
ParticipantsweregiventwoC++programmingproblemstosolvewithin30minuteseach.Eachpromptwasself-containedwithinaCPPfilewithinacompressedfolder,
温馨提示
- 1. 本站所有资源如无特殊说明,都需要本地电脑安装OFFICE2007和PDF阅读器。图纸软件为CAD,CAXA,PROE,UG,SolidWorks等.压缩文件请下载最新的WinRAR软件解压。
- 2. 本站的文档不包含任何第三方提供的附件图纸等,如果需要附件,请联系上传者。文件的所有权益归上传用户所有。
- 3. 本站RAR压缩包中若带图纸,网页内容里面会有图纸预览,若没有图纸预览就没有图纸。
- 4. 未经权益所有人同意不得将文件中的内容挪作商业或盈利用途。
- 5. 人人文库网仅提供信息存储空间,仅对用户上传内容的表现方式做保护处理,对用户上传分享的文档内容本身不做任何修改或编辑,并不能对任何下载内容负责。
- 6. 下载文件中如有侵权或不适当内容,请与我们联系,我们立即纠正。
- 7. 本站不保证下载资源的准确性、安全性和完整性, 同时也不承担用户因使用这些下载资源对自己和他人造成任何形式的伤害或损失。
最新文档
- 2024届九师联盟商开大联考高三第四次模拟考试数学试卷含解析
- 2024年大学计算机基础考试题库含完整答案【名校卷】
- 吉林省吉林市普高2023-2024学年高考压轴卷数学试卷含解析
- 安徽省宿州市时村中学2024年高考数学全真模拟密押卷含解析
- 河北邯郸2024年高三第一次模拟考试数学试卷含解析
- 2023-2024学年湖南省十校共同体高考仿真卷数学试题含解析
- 四川省成都市高中2024届高三第二次联考数学试卷含解析
- 2024届山东省高中名校高三上学期12月调研数学试题和答案
- 2024届高考语文默写回炉练(十一)《短歌行》《梦游天姥吟留别》含答案
- 广东省广州市天河达标名校2024年中考适应性考试化学试题含解析
- 2024年演出经纪人考试题库及答案【典优】
- 水环式真空泵使用说明
- 苏教版小学一年级数学下册期中试卷附参考答案
- 监理工作程序框图
- 战略合作签约仪式教学课件PPT模板
- SBRS交班模板(最新整理)
- 相控阵雷达TR 组件
- 每天工作日志模板
- 融资租赁租金及IRR收益测算表
- 汇川伺服追剪控制指导说明完整版
- 安保科绩效考核指标及量化标准
评论
0/150
提交评论