GPT-4 的代码协助进一步- 利用 GitHub Copilot 和 ChatGPT 进行 VSE 工程的同行评审

EasyChairPreprint

№11192

BeyondCodeAssistancewithGPT-4:LeveragingGitHubCopilotandChatGPTforPeerReviewinVSEEngineering

RoarEliasGeorgsen

EasyChairpreprintsareintendedforrapiddisseminationofresearchresultsandareintegratedwiththerestofEasyChair.

October28,2023

BeyondCodeAssistancewithGPT-4:LeveragingGitHubCopilotandChatGPTforPeerReviewinVSEEngineering

RoarEliasGeorgsen[0000−0003−2868−497X]

UniversityofSouth-EasternNorway,Raveien215,3184Borre,Norway

roar.e.georgsen@usn.no

Abstract.MostcompaniesareVerySmallEntities(VSEs),meaningtheyhavefewerthan25employees.Primarilydomainspecialists,thesecompanieslackin-houseexpertiseinimportantareassuchassecurityandreliabilityengineering,processimprovement,QualityManagement(QM)andSystemsEngineering(SE).VSEsstruggletoadheretoStan-dardOperatingprocedures(SOP),andresearchhasshownthatcon-tractualobligationstofollowindustrystandardsandbestpracticeshavelittleeffectonactualengineering.ThispaperdescribesacasestudythatexploredthepotentialofLargeLanguageModels(LLMs)tosupportengineeringbestpracticesataVSEbytakingontheroleofanexpertpeerinareaswherethecompanyhadaskillsgap.Aiwell,aNorwegianproducerofbuildingautomationequipment,usedChatGPT,GitHubCopilotandGPT-4toassessthequalityoftheirsystemandstakeholderrequirements.AGPT-4foundationmodelwithnoadditionaltrainingwasgivenlinkstoreferencematerialsonrequirementsengineeringpro-ducedbyTheInternationalCouncilonSystemsEngineering(INCOSE)andallowedtoparticipateindiscussionsonthesamedigitalcollabora-tionplatformasthehumanengineers.ThestudyfoundthatAI-assistedrequirementreviewsimmediatelyandpositivelyimpactedtheentireen-gineeringprocess,supportingthefeasibilityofintegratingadvancedAItechnologiesinVSEs,evenwithlimitedtrainingandresources.Partici-pantshighlightedthecomplementarynatureofhumanintelligenceandAI,whereLLMsaugmentedhumanjudgmentthroughdialogue,leadingtoenrichedengineeringpractices.Ethicalanddataprivacyconsidera-tionsalsoemergedascentralthemes,emphasisingtheneedforproactivemeasures.

Keywords:generativeAI·verysmallentities·systemsengineering·re-quirementsengineering

Introduction

TheglobalsupplychainconsistsmainlyofVerySmallEntities(VSE),withaworkforcerangingfromfive(5)totwenty-five(25)people[1],andmicro-enterprises,havingfewerthannineemployees,makeup92%ofEuropeanen-terprises[2].InNorway,VSEsallocateapproximately30%oftheirspendingto

PAGE

10

R.E.Georgsen

BeyondCodeAssistancewithGPT-4

PAGE

11

researchanddevelopment(R&D),andnewproductdevelopmentmakesup20-30%ofVSErevenue,aratesignificantlyhigherthanthelessthan9%observedinlargecompanies[3,4].

Aiwell,asmallNorwegiancompanydevelopingoutdoorautomationsystems,wantedtomoderniseitsengineeringpracticesinresponsetothegrowingcom-plexityofitscustomerprojects.Withahighworkloadandlackofsenioren-gineersinthelabourmarket,Aiwellwantedtoexplorehowtechnologycouldmitigatepotentialskillsgapsandimprovequality.TheseeminglyintuitiveeasewithwhichindividualengineersusedtoolslikeChatGPTandGitHubCopilottogeneratefunctioningcodeanddocumentationmotivatedAiwelltoinitiateapilotstudy.

ObjectivesoftheStudy

ThestudywasdesignedtoexplorethepotentialofLargeLanguageModels(LLMs)toenhanceengineeringpracticesinthecontextofVerySmallEntities(VSEs).Thespecificobjectivesthatguidedthisexplorationwere:

ImpactonProductivityandQuality:InvestigatinghowengineerswithoutpriorexperiencecouldutiliseLLMstoincreaseproductivityandimprovequalityingeneral.

FeasibilityofGenerativeAIforVSEs:AssessingwhetherLLM-basedtoolscouldbeintegratedintoaVSEworkflowatanaccessiblecostandwithoutadditionalhumanresources.

ImpactonEngineeringCompetency:InvestigatinghowLLMscouldfillknowl-edgegapsandcontributetoincreasedcompetencyinVSEs,inparticularwithregardtorequirementsengineering.

Byfocusingonthesekeyareas,thestudyintendedtoprovideinsightsintohowLLM-basedtoolscanaddressspecificchallengesfacedbyVSEs.

Background

EngineeringinVerySmallEntities(VSE)

VSEslackexperienceworkingwithstandardisedprocesses[5]andoftenleantowardsinformalandorganicallyevolvedmethodologies.Theyfindstandardisedmethodstoobroadfortheirspecificneedsandvaluetheagilityandtheabilitytotailorworkflowsthatinformalityofferstosuittheiruniquecontexts[6].Smallcompaniesareconsciousoftheescalatingcustomerandlegalexpectationsforsystematicengineeringandtheirinternalneedforimprovement[5].However,theydonotviewtheseasbeneficialtotheirworkorrelevanttotheirsituation[6].Thisbeliefintheabsenceofaddedvalueleadstooppositiontochange,

andtheactualengineeringpracticesmaynotalignwithdocumentedcompliance[7].VSEsarerarelyrequiredtodocumentcompletecompliancewithspecificstandardsandthusprefertoproduceonlytheminimumdocumentationrequiredbycontractualobligations[6].Measuressuchasrisk-sharingpartnerships[8]intendedtoimprovesystemqualitycan,inthiscontext,reducetheactualqualityofthesystemasresponsibilityandaccountabilitymovedownthesupplychain.StandardizingengineeringpracticescanbevitaltoVSEsforamultitudeofreasons.ThesepracticesequipVSEswiththeresourcesandprovenmethodolo-giestoenhancethequalityandefficiencyofengineering.Theyfosterimprovedprojectmanagement,reinforceimplementationprocesses,andcontributetocom-petitiveness[5].TheISO/IEC29110seriesandsimilarinitiatives,tailoredforVSEs,canfacilitatethetransferofcodifiedknowledgerelatedtosystemsengi-neering,offeringbenefitssuchaspromotinginnovation,marketaccess,quality

control,andethicaladherence.

Standardscanprovidelegalprotection,shieldingengineersfromaccusationsofnegligence.Thefollowingscenarioin[9]involvingan8-personcompanythatdevelopedcomputer-controlledvalvesforindustriessuchaspharmaceuticalsandchemicalsillustratesthecriticalimportanceofstandardizedengineeringprac-tices,evenforVSEs.Thecustomercontractedthecompanytoinspectthere-quirementsusingIEEEsoftwareengineeringstandards.However,thedeveloperwasunawareoftheIEEE1028standard,whichdescribesthetypesofsoftwarereviewsandproceduresrequiredforexecution.Afterinstallingthenewsoftware,thecomputer-controlledvalvesmalfunctioned,causingdamagesinthechemicalplantandleadingtolegalaction.Thecourthearingrevealedthatthesuppliercouldnotprovideevidenceofaninspectionaccordingtothestandard,resultinginlegalandfinancialconsequences.Thisincidentunderscoresthenecessityofadheringtostandardizedengineeringpracticestoensurequality,safety,andle-galcompliance,regardlessofthecompany’ssize.Italsohighlightsthepotentialrisksandliabilitiesthatcanarisefromneglectingthesestandards.Inessence,standardizedengineeringpracticescanserveasacornerstoneforVSEs,ensuringquality,legalcompliance,andeconomicviability,therebycrucialtotheirsuccessandsustainability.

RequirementsandSystemsEngineering

InAiwell’sendeavourtoimproveitsengineeringpractices,requirementsqual-ityemergedasapivotalfocusarea.Theimportanceofrequirementsqualityismulti-fold.First,well-definedrequirementsserveasthecornerstoneforsuccessfulprojectoutcomes,reducingthelikelihoodofcostlyrevisionsanddelays.Second,theyfacilitateclearcommunicationamongstakeholders,ensuringalignmentonobjectivesandexpectations.Thereisastrongpositivelinkbetweencapabilityinrequirementsdefinitionandmanagement[10].Ahigh-qualityrequirementisclear,concise,verifiableandtraceable.Itshouldbedevoidofambiguity,allowingforasingleinterpretation,andverifiablethroughinspection,analysis,ortesting.Critically,arequirementshouldalsobetraceable,linkingbacktoitssource,ra-

tionale,anddependencies,whichaidsinmanagingchangesandunderstandingimpacts.

LargeLanguageModels(LLMs)

LargeLanguageModels(LLMs)areasubsetofartificialintelligencemodelsdesignedtogeneratehuman-liketext.Thesemodelshavegainedsignificantat-tentioninsoftwareengineering,particularlyinautomatingcodingtasksandim-provingcodequality.LLMshavebeenevaluatedfortheireffectivenessincodegenerationandhaveshownpromisingresultsinvariousaspectsofsoftwarede-velopment[11].Thestudyrevealedthat85%ofdevelopersfeltmoreconfidentintheircodequalitywhenauthoringwithGitHubCopilot.Moreover,codere-viewswerecompleted15%faster,and88%ofdevelopersreportedmaintainingaflowstate,indicatingincreasedfocusandreducedfrustration.LLMsarenotjustacceleratingthecodingprocessbutarealsoenhancingthequalityofthecode.GitHub’sinternalmetricsforcodequality—readability,reusability,con-ciseness,maintainability,andresilience—showedsignificantimprovementwhendevelopersusedGitHubCopilot[12].

Usingartificialintelligence,andmorerecentlyDeepLearning,tosupportengineeringisnotnew[13],butinlearningtouselanguage,LLMsarealsopickingupotherhumanskills,suchaslearningfromasingleexample[14].WhereasnotlongagotraininganAIwasanexpensiveprocessthattookalongtime,LLMfoundationmodelscanlearnbyjustlookingupthedocumentation[15].ItisthislastnewfoundabilitythatAiwellaimedtoexploitintheirpilotstudy.

CaseStudy:LeveragingLLMstoEnhanceRequirementsReviewsinaVSE

CaseDescription

Aiwellisacompanywithsevenemployeesproducingsoftwareandhardwareusedinbuildingautomationsystems.Duetohighworkloads,Aiwellfounditchallengingtoallocatesufficienttimeonaconsistentbasistodefinehigh-qualityrequirementspecificationsforitsdiverserangeofprojects.ThecompanywantedtoleverageLLM-basedtoolstoautomateandenhancethequalityofstake-holderandsystemsrequirements,guidedbytheINCOSESystemsEngineeringHandbook[16]andtheSystemsEngineeringBookofKnowledge(SEBoK)[17].EngineersatAiwellalreadyusedGitHubforversioncontrolandwantedtoexploreGitHubActionsfortaskautomation.TheyemployedChatGPTPlusandGitHubCopilotforAI-assistedscriptdesignandtheOpenAIGPT-4APIinscriptexecution.Keymetricsincludedthenumberofrequirementrevisions,timespentoneachrequirement,andthefrequencyofrequirement-taggedissuesapprovedwithoutfurtherrevisions.Initialimplementationledtomorepreciseandcompleterequirements,withfewerrevisionsandreducedtimespentoneachrequirement.Thissectionprovidesdetailsandpracticalinsightsbydemonstrat-inghowAiwellusedLLMseffectivelyandefficientlyregardingresourcesandtime

toimprovetheirengineeringpractices,inparticularwithregardstoimprovingrequirementsquality.

Methodology

Thestudyemployedtwoprimarymethodologies:ActionResearchandGlaser’sGroundedTheoryMethod(GTM).

ActionResearch(AR)isaparticipatory,iterativemethodologyfocusedonsolvingreal-worldproblemsthroughcyclesofplanning,action,observation,andreflection[18].AiwellconductedmultiplecyclesofActionResearchtorefinetheapproach,resolvingimmediatechallengesandplanningforlong-termadaptabil-ity.

Glaser’sGroundedTheoryMethod(GTM)isaqualitativeresearchap-proachthatemphasisesthegenerationoftheorydirectlyfromdatathroughiterativecodingandconstantcomparativeanalysis[19].GTMwaschosenforitsflexibilityinhandlingqualitativedata.Thestudybeganbyapplyingopencodingtorawelectroniccommunication,includinginstantmessagingandcom-mentthreads.Thispreliminaryanalysisidentifiedinitialconcepts,whichwerecontinuouslyrefinedintocategoriesandthemes.Thestudyadoptedaconstantcomparativeanalysis,integratingnewdataiterativelytoevolvetheemergenttheory.Thisapproachmadesuretheresultingtheorywasbothrelevantandcontextuallygrounded.

EthicalandCommercialConsiderations

GivenAiwell’sstatusasaVerySmallEntity(VSE),thestudyhadtoaccountforethicalconcernsassociatedwithdataconfidentialityandanonymity.StrategieswerebasedonSAGEguidelines[18],andincludedfocusingoncollectiveinsightstopreserveinternalanonymityandworkingcloselywithAiwelltoredactsensi-tiveorproprietarydata.

AWorkingDefinitionofRequirementsQuality

Toquantitativelyassessthequalityofarequirement,Aiwellemployedascor-ingsystembasedonkeyattributessuchasclarity,conciseness,testability,andtraceability.Eachattributewasassignedaweight,andallrequirementswereevaluatedonascaleof1to5foreachattribute.Aggregatingtheattributescoresproducedacompositequalityscore.Forexample,arequirementwithaclarityscoreof4,concisenessscoreof5,testabilityscoreof3,andtraceabilityscoreof4wouldyieldacompositescoreof(4*0.3)+(5*0.15)+(3*0.35)+

(4*0.20)=3.8.

EngineersevaluatedrequirementsusinganapproachbasedonPlanningPoker[20],anagileestimatingtechniquewhereteammembersuseplayingcardstovoteonthecomplexityofatask,facilitatingdiscussionandconsensus.

Requirementsengineeringbestpracticesweredrawnfromtwoseminalre-sources:”TheINCOSESystemsEngineeringHandbook”and”TheSystemsEn-gineeringBookofKnowledge”(SEBoK).TheseresourceswerealsoaccessibletotheLLMsthroughhyperlinks,aswasawrittendescriptionofthesystemcon-textandthescoringsystemusedbyhumanevaluators.Thisprovidedabasicframeworkforassessingrequirements.

IntegratingLLMsintheWorkflow

IntroducingLLMsintotheengineeringworkflowhadtobelow-costandrequireminimaltraining.ChatGPTandGitHubCopilotservedastheprimaryLLM-basedtoolsforthispurpose.ChatGPTisanAIconversationalagentcapableofgeneratinghuman-liketextbasedonthepromptsitreceives.GitHubCopilotisanAI-poweredcodeassistantthathelpsengineerswritenewcodeandun-derstandandworkwithexistingcodemoreefficiently.Thetools’affordabilitymadethemattractivechoicesforabudget-conscioussmallentity.GitHubCopi-lot’sseamlessintegrationwithMicrosoftVisualStudioCode,afreecodeeditoralreadyusedbyAiwell,facilitatedasmoothtransition.TheOpenAIGPT-4API,thoughdemandingasteeperlearningcurve,wasrenderedmoreacessiblewiththeassistanceofChatGPTandGitHubCopilot.Whilethecodegener-atedbythesetoolsmightnotalwaysbeperfect,itwassufficienttoexpeditethedevelopmentprocesses.LLMscouldalsoparticipateinmulti-lingual,realis-ticandnuancedhuman-likediscussionswithengineersaboutsubtletopicslikerequirementquality.

Initially,AiwellengineersstartedwiththedefaultinterfacesofChatGPTandGitHubCopilot.Thesetools’easeofuseandintuitiveinterfacemeantthatengineerscouldbeginleveragingtheircapabilitieswithoutextensivetrainingorpreparation.Thisfactorwascrucial,giventheresourceconstraintstypicalofVSEs.Theuser-friendlyandadaptabletoolsfiteffortlesslyintoAiwell’sexistingGitHub-basedversioncontrolandtaskautomationworkflows.

WhenOpenAIintroducedcustominstructionsasanewfeatureinChatGPT,Aiwellincorporateditintotheirworkflowbyaddinglinkstorelevantreferencesasstandardprefixestotheirprompts.ThisallowedengineerstoguidetheAImodelmoreeffectively,aligningthegeneratedresponseswiththeproject’sspe-cificcontextandneeds.StandardisingcustominstructionenhancedthequalityofLLMresponsesbymakingthemmorepreciseandcontext-appropriate.

Astheteamgainedexperience,theybeganexploringmoreadvancedfea-turesoftheGPT-4API.TheystartedcraftingpromptsthatcommunicatedtheirintentmoreclearlytotheLLMs,improvingthequalityofthegeneratedresponse,whethercodeorrequirementsevaluation.Thisincrementalandorganicdeveloper-ledapproachtoadoptingLLMfeaturesensuredtheteamcouldadaptwithoutfeelingoverwhelmed,therebymaintainingproductivity.

LLMsbecameanintegralpartofAiwell’sengineeringpractices,notasadisruptivetechnologyrequiringasteeplearningcurvebutasenablersthatwereincrementallyadopted.Themetrics,includingthereducedrequirementrevisionsandtimespentoneachrequirement,validatedtheeffectivenessofintegratingLLMsintotheworkflow.Crucially,thesegainsneverrequiredadvancedappli-cationssuchaspre-trainedAImodelsorcuratedvectordatabaseswithcustomknowledgebutreliedonlyonfreeandlow-cost,publicallyavailabletools.

CollaboratingWithLLMsonWritingAutomationScripts

Aiwell’sengineersleveragedChatGPTtoautomateGitHubActions,focusingonrequirementsvalidation.TheprocessbeganwithasimpleprompttoChat-GPT,askingittodraftaGitHubActionscripttocalltheGPT-4APIifauserlabelledaGitHubrepositoryissueasarequirement.DespitesomeinitialregularexpressionandJSONparsingchallenges,theengineersiterativelyrefinedtheprompts,leadingtoeffectivescripts.Figure1showsanabbreviatedexampleofthegeneratedcode.

GPT-4’soutput,postedasaGitHubcommentontheissue,comprehen-sivelyevaluatedtherequirementusingSEBoKandtheINCOSEHandbookasareferenceandprovidedtheengineerwithatasklistofsuggestedimprove-ments.Figure2showsthefullanalysisofarequirementgeneratedbyGPT-4.Lateriterationsincludedacompositescorebasedontheprovidedguidelines.TheAI-generatedcommentwouldbreakdowntherequirement’sclaritybyas-sessingitsspecificity,concisenessbyindicatingunnecessarydetails,testabilitybyevaluatingtheclearnessofacceptancecriteria,andtraceabilitybycheckingitslinkagetosystemneedsorstakeholderrequirements.Eachattributewouldreceiveascore,andGPT-4wouldcalculateacompositescoreusingthesamescoringsystemandguidelinesemployedbyAiwell’shumanengineers.Thisap-proachenrichedtherequirementsvalidationprocess,offeringaquantitativeandqualitativeassessmenttosupplementthehumanscoringandencouragecriticaldiscussion.ItalsoshowedhowVSEs,evenwithlimitedresources,canincremen-tallyintegrateLLMsintotheirexistingworkflows.TheuseofChatGPTinscriptautomationnotonlystreamlinedthetaskbutalsoaddedalayerofintelligenceandreview,makingtheprocessmorerobustandefficient.ThispartofthestudydemonstratedthecapabilityofLLMbasemodelstoperformcomplexsystemsengineeringtasksthatalignwithestablishedsystemsengineeringprinciples.

1name:CheckRequirementCorrectness

2on:

3 issues:

4

types:[labeled]

5jobs:

6 check-requirement:

7 runs-on:ubuntu-latest

8 if:contains(github.event.label.name,’requirement’)

9 steps:

10 -name:Checkoutcode

11 uses:actions/checkout@v2

12 ...

13

14

-name:Check

env:

requirement

15 GITHUB_TOKEN:${{secrets.MY_GITHUB_TOKEN}}

16 OPENAI_API_KEY:${{secrets.YOUR_OPENAI_API_KEY

}}

17 run:|

18 ...

19 #LoadtheStatementofWork

20 SYSTEM_CONTEXT=$(catREADME.md)

21 SCORING_GUIDELINES=$(catscoring.md)

22

23 #Getthecontentrelatedtocheckingcorrectnessofsystemrequirements

24 CORRECTNESS_GUIDELINES="https://sebokwiki.org/wiki/Stakeholder_Requirements_Definition#Checking_Correctness_of_System_Requirements"

25 ...

26 #CallGPT-4APIwithissuedetails

27 RESPONSE=$(curlhttps://api.openai.com/v1/chat/

completions\

28 -H"Content-Type:application/json"\

29 -H"Authorization:Bearer$OPENAI_API_KEY"\

30 -d’{

31 "model":"gpt-4",

32 "messages":[

33 {"role":"system","content":"

$SYSTEM_CONTEXT

$CORRECTNESS_GUIDELINES

$SCORING_GUIDELINES"},

34 {"role":"user","content":"Title:

$ISSUE_NAME.Id:$ISSUE_NUMBER.

Description:$ISSUE_BODY"}

35 ]

36 }’)

37 ...

38 #AddtheresponseasacommenttotheissueusingGitHubAPI

39 curl-s-XPOST-H"Authorization:token

$GITHUB_TOKEN"-H"Content-Type:application/json"-d"$PAYLOAD""$COMMENT_URL"

Fig.1.PartialGitHubActionymlscriptgeneratedbyChatGPT

Thetemperatureoftheglycolmustnotexceed50degrees.##AIGeneratedEvaluation

Therequirementisclear,concise,andunambiguous.However,it

couldbeimprovedbyprovidingmorecontextandspecificity.

Whileitstatesaspecifictemperaturelimit,itdoesnotspecifytheconditionsunderwhichthislimitapplies(e.g.,duringoperation,atrest,etc.).Also,itdoesnotprovideanyacceptancecriteriaoramethodfortestingthisrequirement.

Here’satasklistforimprovingthisrequirement:

[]Specifytheconditionsunderwhichthetemperaturelimitapplies.Forexample,"Thetemperatureoftheglycolmustnotexceed50degreesCelsiusduringoperation."

[]Definetheacceptancecriteria.Forexample,"Thesystemisconsideredtomeetthisrequirementiftheglycoltemperatureremainsbelow50degreesCelsiusduringacontinuous24-houroperationunderstandardsnowfallconditions."

[]Provideamethodfortestingthisrequirement.Forexample,"Thisrequirementcanbetestedbyoperatingthesystemcontinuouslyfor24hoursunderstandardsnowfallconditionsandmonitoringtheglycoltemperature."

Aftertheseimprovements,therequirementmightlooklikethis:

Thetemperatureoftheglycolmustnotexceed50degreesCelsiusduringoperation.Thesystemisconsideredtomeetthisrequirementiftheglycoltemperatureremainsbelow50degreesCelsiusduringacontinuous24-houroperationunderstandardsnowfallconditions.Thisrequirementcanbetestedbyoperatingthesystemcontinuouslyfor24hoursunderstandardsnowfallconditionsandmonitoringtheglycoltemperature.

Fig.2.GPT-4generatedevaluationofrequirement

Challengesandsolutions

DataPrivacyandSecurityIntegratingGitHubCoPilot,ChatGPT,andtheGPT-4APIintoAiwell’sworkflowpresentedsignificantsecurityandprivacychallenges,primarilyduetoalackofclearpoliciesandconcernsexpressedbygovernmentagencies[21].Tomitigatethisuncertainty,thetoolswereinitiallyrestrictedtonon-sensitivematerial.However,asthestudyprogressed,theintro-ductionofGitHubCoPilotforBusinessandMicrosoft’sAzurehostedversionsofOpenAI’smodelsprovidedmoresecurealternatives[22].OpenAI’salsoclarifiedinitspoliciesthatGPT-4APIdataisnotstoredbytheirservers,alleviatingsomeconcerns[23].

ModelLimitationsWhileChatGPTandGPT-4demonstratedproficiencyindomain-specificlanguage,themodelsfalteredintaskslikeregularexpres-sionparsing,mathematicsandJSONformatting.OpenAI’ssubsequentupdate,whichintroducedfunction-callingcapabilitiesintheGPT-4API,addressedtheseissues,enablingdeterministicfunctionsforcomplextasks.

Human-AICollaborationBalancinghumanandAIcontributionsprovedchallenging.AlthoughChatGPTgeneratedeffectiveYMLscripts,itsnumeri-calrequirementscoresoftendivergedfromhumanevaluations.ActionResearchmethodologyhelpedhere,asdiscrepanciestriggereddiscussions,leadingtoanevolutionoftheevaluationcriteriaandamorestableconsensusamongengineers.Also,whengivenaccesstothecommentssectiononGitHub,GPT-4wouldjointhediscussionandbeasopinionatedasahumanengineerifinstructedspecifi-callytobeso.

QualityControlEnsuringthequalityofAI-generatedevaluationsrequiredvigilantoversight.Asystemsengineerconsistentlyreviewedthemodel’soutputs,maintainingahuman-in-the-loopapproachatalltimes.Thisreviewprocesswasintegraltotheiterativecyclesofplanning,action,observation,andreflection.

ErrorHandlingScriptsgeneratedbyChatGPTandGirhubCo-Pilotun-derwentrigorousscrutinyandtesting,andengineersprovidedthetoolswithpromptsbasedonatemplatethatevolvedincrementallybasedonpreviousmistakesmadebytheLLMs.However,sinceGPT-4’soutputwaslimitedtocommentsandnotexecutablecode,theriskofoperationaldisruptionswasmin-imised.

ScalabilityScalingtheapproachforlargertasksorteamsposedchallenges.Thesolutioninvolvedusingsmall,template-basedscriptsandleveragingLLMsforextensivecommentinganddocumentation.

Insummary,thechallengesencounteredbyAiwellweresystematicallyad-dressed,oftenbenefitingfromtheiterativeandreflectivenatureoftheAction

Researchmethodology.Thisapproachnotonlyresolvedimmediateissuesbutalsocontributedtothelong-termadaptabilityandrobustnessoftheengineeringworkflow.

EmergentThemes

TheGroundedTheoryMethod’siterativenaturewaspivotalinthecontinuousrefinementandevolutionoftheemergentthemes.ByensuringthatthesethemesweredeeplyrootedintheexperiencesandfeedbackofAiwell’sengineers,thestudycapturedtheauthenticchallengesandopportunitiesofintegratingAIintotheworkflowsofaVSE.Severaldistinctthemesemergedfromthestudy,eachsheddinglightondifferentfacetsofintegratingAIintotheworkflowsofasmallcompanylikeAiwell.ThesethemesprovideinsightsintotheimmediatebenefitsandchallengesofAIadoptionandhintatthebroaderimplicationsforthefutureofengineeringpractices.

AccessibilityTheeaseofintegratingAItoolsintoAiwell’sworkflowsunder-scoredthethemeoffeasibilityandaccessibility.SomeengineershadinitiallyperceivedtheadoptionofAIasadauntingtask.However,theuser-friendlynatureoftoolslikeChatGPTandGitHubCopilotfacilitatedasmoothtransi-tion.AcommentfromanAiwellteammembercapturedthissentiment:”WethoughtintegratingAIwouldbeamassiveundertaking,butthesetoolsmadethetransitionsurprisinglysmooth.”Thisthemeemphasisesthedemocratisationofadvancedtechnologies,makingthemaccessibleeventosmallerentities.Intheearlystagesofthestudy,thedatapointedtowardsthefeasibilityofintegratingAItoolsintoAiwell’sworkflows.However,asengineersgainedmorehands-onexperiencewiththesetools,theirfeedbackbegantoreflectabroaderperspective.Commentslike”Theintegrationwassmootherthanweanticipated”highlightedthefeasibilityandaccessibilityofadvancedAItechnologies.Thisdevelopmentunder