ARCeR：用于自动定义网络靶场的代理 RAG

ARCeR:anAgenticRAGfortheAutomatedDefinitionofCyberRanges

MatteoLupinacci1[0009−0000−2356−398X],

FrancescoBlefari1,2[0009−0000−2625−631X],

FrancescoRomeo1,2[0009−0006−3402−3675],

arXiv:2504.12143v1[cs.CR]16Apr2025

FrancescoAurelioPironti1[0009−0003−3183−2977],and

AngeloFurfaro1[0000−0003−2537−8918]

1UniversityofCalabria,87036,Italy

{matteo.lupinacci,francesco.blefari,francesco.romeo,francesco.pironti,angelo.furfaro}@unical.it

2IMTSchoolforAdvencedStudies,Lucca,55100,Italy

Abstract.Thegrowingandevolvinglandscapeofcybersecuritythreatsnecessitatesthedevelopmentofsupportingtoolsandplatformsthatal-lowforthecreationofrealisticITenvironmentsoperatingwithinvir-tual,controlledsettingsasCyberRanges(CRs).CRscanbeexploitedforanalyzingvulnerabilitiesandexperimentingwiththeeffectivenessofdevisedcountermeasures,aswellasservingastrainingenvironmentsforbuildingcybersecurityskillsandabilitiesforIToperators.ThispaperproposesARCeRasaninnovativesolutionfortheautomaticgenerationanddeploymentofCRs,startingfromuser-provideddescriptionsinanaturallanguage.ARCeRreliesontheAgenticRAGparadigm,whichallowsittofullyexploitstate-of-artAItechnologies.Experimentalre-sultsshowthatARCeRisabletosuccessfullyprocesspromptsevenincasesthatLLMsorbasicRAGsystemsarenotabletocopewith.Fur-thermore,ARCeRisabletotargetanyCRframeworkprovidedthatspecificknowledgeismadeavailabletoit.

1Introduction

Inrecentyears,theincreasingnumberofthreatstargetingITsystemshasledtoasignificantfocusinthecybersecurityresearchcommunityonthedevelop-mentofCyberRanges(CR)asfundamentaltoolstotrainITprofessionalsinfacingcyberthreatsandattacks.Typically,settingupaCRinvolvesdefiningitsinfrastructureandsoftwareinaconfigurationfile.ACRplatformadmin(suchascybersecurityinstructors)hastomanuallydesign,buildanddeploycustomscenariosbywritingthesefiles,anactivitywhichisbothtimeconsum-ingandprone-to-error.Inaddition,thecomplexityofbuildingandmaintainingreal-worldscenarios,bothinenterpriseandeducationalsettings,ischallenging.

Inthelastfewyears,CRframeworks,asindicatedin[1]and[2],havebe-comeawidespreadsolutioninassistingtheCRdevelopmentlife-cycle.Severalframeworkshavebeenproposed,eachwithdistinctfeaturesandobjectives,that

2Lupinaccietal.

mainlyfocusonone(orboth)ofthefollowingtwomainaspects:(i)enhancethegenerationofincreasinglysophisticatedtrainingscenariosthataccuratelyreflectreal-worldthreatsand(ii)reducethetimeandresourcesnecessaryfortheconfig-urationandexecutionofCRinstances(seeSection6).Thelattermeansthattheframeworkautomaticallyprovidesthenecessaryinfrastructureandestablishesconnectionsbetweenmachinesaccordingtotheirnetworktopologywithouttheinterventionoftheinstructor.

Theuseofmachinelearning(ML)toimproveCRdeploymentisexploredin[3],whereasystembasedontheOpenStackcloudplatformispresentedtoautomateCRdeployment.ThissystemusesMLtoclassifyVMstoreducethecostofmanualselection.Withthatplatform,theinstructoronlyneedstoprovideaYAMLdescriptionofavirtualenvironmentthatisparsedtocompletetheautomateddeployment.

Amongthemoreinnovativeapproaches,theworkby[4]standsoutasthefirsttoproposetheapplicationofLLMtotheCRdomain.TheirapproachtransformsthewellknownandinherentinLLMs‘hallucination’problemintoapotentialadvantage,allowingthecreationofcomplexscenariosthatpushtheboundariesoftraditionalcybersecuritytraining.

Tothebestofourknowledge,thisisthefirstworkwhichproposestheuseofLargeLanguageModels(LLMs)andAgenticRetrieval-AugmentedGenera-tion(AgenticRAG)[5]systemsfortheautomateddefinitionanddeploymentoftrainingscenarioscompatiblewithmultipleCRplatforms.

ThedevisedapproachaimstosimplifythisprocessbygeneratingCRfromnaturallanguagedescriptions.TheabilityofAgenticRAGstoplan,usetoolsexternaltotheLLM,dynamicallyadapttoresponsesfromtheenvironment,andperformmultipleretrievalstepsfromexternal-supplieddocumentsenablesthemtoreducethecostsrequiredforfine-tuninganLLMandadapttomultipleCRinstantiationplatforms.

Furthermore,theuseofaugmentedknowledgeempowerstheagenttogener-atevalidconfigurationfilesforagivenCRplatform.Thisrepresentsasignificantadvancecomparedtopreviousworks[3,4,6]thatemployedspecifictechniquesfortheautomaticgenerationofrandomizedscenarios.Insummary,themaincontributionsofthispaperare:

–theproposalofanovelapproachbasedonAgenticRAGsystemsfortheautomatedinstantiationofCRfromnaturallanguagetextdescriptionsoftheinfrastructurethatiscompatiblewithmultipleCRplatforms;

–thesupportfortheautonomousgenerationofself-devisedCRscenarioslever-agingAgenticRAGsystems;

–thedesignandimplementationofARCeR,thefirstAgenticRAGsystemforCRdefinitionanddeployment(thistoolwillbemadeavailableopen-sourcetofosterfutureresearchinthisarea).

TheeffectivenessofARCeRhasbeenassessedbytestingitwithCyRIS[7]awell-knownCRframework.IthasbeenobservedthatwhilethegenerationoffilesbasedonspecificknowledgecangenerallybeaccomplishedwithapureRAG

ARCeR:anAgenticRAGforCyberRanges3

system,employinganAgenticRAGcanenhancetheaccuracyandintegrityoftheoutputwhilemaintainingequivalentcomplexity.

Theremainderofthispaperisorganizedasfollows.Section2providesthebasicbackgroundonCyberRangesand(AI)Agents.Section3presentstheproposedapproachtotheautomaticinstantiation,generationanddeploymentofCRusingAgenticRAG.Section4presentsacasestudybasedonCyRIS.Section5discussestheachievedresults.Section6overviewstherelatedworksandSection7drawstheconclusions.

2Background

2.1CyberRanges

TheUSNationalInstituteofStandardsandTechnology(NIST)definesCyberrangesasinteractive,simulatedrepresentationsofanorganization’slocalnet-work,system,tools,andapplicationsthatareconnectedtoasimulatedInternetlevelenvironment.Theyprovideasafe,legalenvironmenttogainhands-oncy-berskillsandasecureenvironmentforproductdevelopmentandsecurityposturetesting[8].Cyberrangesarecontrolledandinteractivevirtualenvironmentsthatareutilizedforefficientandsecurecybersecuritytraining,aswellasforthese-cureandcontrolledemulationofnewreal-worldattacksandmalware.Forthisreasons,theycanbeconsideredascontemporarybattlefieldsforcybersecurity.Moreovertheyoffertraineesacontextforaugmentingtheircybersecurityskillsetthroughhands-onactivities(suchasthereal-timeanalysisframeworksforCRsasproposedin[9,10]).

ACRcomprisesanITinfrastructureandasuiteofselectedsecurityfeatures.

Theinfrastructurecaninclude,butisnotlimitedto,machines,networks,storage,andsoftwaretools.Securityfunctionalitiesincludethecapabilitytoreplicatecy-berattacksandexecutemalwareinacontrolledenvironment.TheorchestrationlayeroftheCRcoordinatesthediversetechnologyandservicecomponentsoftheunderlyinginfrastructureandprovidesisolationfromotherresourcesonthehostsystems.Thisisolationenablesthesimulationofcomplexscenarioswith-outcompromisingliveproductionsystems.ACRmayalsointegrateaLearningManagementSystem(LMS)thatallowsbothinstructorsandtraineestotrackandmeasureprogressthroughadefinedtrainingcurriculum[11].

TheutilizationofCRshasundergoneasubstantialtransition,withashiftfromtheirinitialadoptionbymilitaryandgovernmentagenciestotheircurrentapplicationbyawidearrayofbusinessesandorganizationsincludingbug-bountyhunters,researchersandstudents.Notably,CRshavefoundapplicationinallow-inginasecureway,thedynamicanalysisofmalwareutilizedintargetedattacks,wheretheexecutionofthemaliciouscodeisnecessarytodetermineitspurpose.

AccordingtotheNIST,therearefourmaincategoriesofCRs:(i)simulations,(ii)overlay,(iii)emulation,and(iv)hybridranges.Thesedistinctionsassumeparticularsignificancewhenalignedwiththespecificusecaseofanindividualororganization[12].

4Lupinaccietal.

Simulationsrangesentailtheestablishmentofasyntheticnetworkenviron-mentthatsimulatesthebehaviorofrealnetworkcomponentswithinvirtualinstances(VMs).VMsareusedtomimicspecificserversornetworkofvariousinfrastructures.Theyoffertheadvantageofquickreconfigurationbecausetheyusestandardizedtemplates.However,thefidelityoftheexerciseincreasesasthesimulationcloselymatchesthetargetinfrastructure.Nevertheless,theunpre-dictableandunrealisticlatencyandjitterinnetworkperformancearefactorsthatcanimpacttheoverallrealismofthesimulationsotheyshouldbelimitedtotheleastpossibleextent.

Overlayrangesprovideahigherleveloffidelityastheydirectlyutilizetheactualnetworkinfrastructure.However,thisincreasedfidelitycomeswithno-tablecostsforhardwareandthepotentialriskofcompromisingtheunderlyingnetworkinfrastructure.ThistypeofCRsareoftenestablishedasglobaltestbedsforresearchandexperimentation.

EmulationisanapproachtoCRgenerationthattransformsthephysicalin-frastructureintothecyberrangeitself.Itprovidesclosed-networkenvironmentsthatconsistofmultipleinterconnectedcomponentsandincludestrafficgenera-tionthatemulatesvariousprotocols,sourcepatterns,trafficflowsandattacks.Emulationgivestraineetheauthenticexperience,ratherthanpre-programmedactions.AnotableexampleoftheuseofemulationinCRistheNationalCyberRange(NCR)[13].

Hybridrangesareformedthroughacustomizedcombinationofanyofthepreviouslymentionedtypesthatsuitsspecificrequirements.Aprominentexam-pleofHybridrangesistheEuropeanFutureInternetResearch&Experimenta-tion(FIRE)project.[14]

2.2Agents

AnAgent[15]isdefinedasacomputersystemsituatedinanenvironmentthatiscapableofactingautonomouslyinitscontextinordertoreachitsdelegatedobjectives.

Autonomymeanstheabilityandrequirementstodecidehowtoacttoachieveagoal.Anagentthatcanperceiveitsenvironment,reacttochangesthatoccurinit,taketheinitiative,andinteractwithothersystems(likeotheragentsorhumans)iscalledanintelligentagentorAIAgent(Fig.1).AnothercoreconceptofAIagentsisthememory.Effectivememorymanagementimprovesanagent’sabilitytomaintaincontext,learnfrompastexperiences,andmakemoreinformeddecisionsovertime.

Aspointedoutin[16],theemergenceofLLMsrepresentsanothermomentofprogressintherealizationofAIagents.ThesubstantialadvancementsmadeinthisdirectionhaveculminatedintheemergenceofLLMagents.Inparticular,LLMagentsuseLLMsasreasoningandplanningcorestodecidethecontrolflowofanapplicationwhilemaintainingthecharacteristicsoftraditionalAIagents.LLMagentsenhancedtheLLMsbyallowingthemtoinvokeexternaltoolstosolvespecifictasks,suchasmathematicalcalculationsorcodeexecution.Inthe

ARCER:anAgenticRAGforCyberRanges

Pereeption

Environment

Lookatheaky

doyouthinkit

willraintomorow?Ifso,givethe

umkcellatome

Reaocnngfrom

thecurrentweatherconditioesandtheweatherrportsontheintemet,itis

likelytonin

tomorrow.Hereisyourumbrela.

/Reasoning

Retrieve

Fig.1.AIagentstructure[16]

end,theLLMcandecidewhetherthegeneratedanswerissufficientorifmoreworkisneeded.

2.3AgenticRAG

Despitetheirflexibility,generalpurposeLLMs,andtherelatedLLMagents,oftenlackthedomain-specificknowledgerequiredtosolveparticularandnon-trivialtasks.Suchaproblemcouldbesolvedbyre-trainingorfine-tuningthemodel;however,thecostoftheseoperationsisnotnegligible.

ApossiblealternativeapproachisbasedontheRetrieval-AugmentedGen-eration(RAG)[17]paradigmwheretheLLM,inpursuingitsgoal,cooperateswithtwoexternalcomponents:(i)asourceofdomain-specificknowledge(i.e.external-supplieddocuments)and(ii)aretrieverwhichisinchargetosearchforrelevantinformationintheexternalknowledgebaseandtoaugmentthecontextoftheLLM.Thisoperationprovidesthemodelwhatisneededtoan-swercomplexquestions.InRAGsystems,externaldataareloadedanddividedintochunksoftheappropriatesize.Thechunksarethenconvertedintovectorrepresentationsandstoredindatastructuresforfutureuse.ThefunctioningofaRAGsystemistypicallystructuredasfollows:(i)theuserprovidesaquerytothesystem;(ii)theretrieverconvertsthequeryintoavectorrepresentationandperformsamatchwiththestoredembeddings,fetchingthemostrelevantchunks;(iii)theoriginalqueryisenhancedwiththefetchedchunksandpassedtotheLLM;(iv)thecontext-awareresultsproducedbytheLLMarereturnedtotheuser.

AnagentsystemthattakesadvantageoftheRAGparadigmisoftenreferredtoasAgenticRAG.Manystate-of-the-artframeworks,suchasLangChain[18]Llamalndex[19],andLangdroid[20],provideeasy-to-useinterfacestocreatecustomAgenticRAGs.

6Lupinaccietal.

3ARCeR:AgenticRAGforCyberRanges

MostCRplatformsreceiveasinputforCRgenerationanddeploymentade-scriptionfilecontaininginformationaboutthehostrunningtherangeandtheCRcharacteristics(e.g.,thevirtualmachinesthatmakeuptherangeandthenetworktopology)withthecorrespondingimagefiles.Descriptionfilesarewrit-teninspecificformatssuchasYAML[21]and,dependingontheCRplatformandrangespecification,theycanbecomeverycomplextowrite.TraditionalCRinstantiationmethodsrequiremanualconfiguration.

ARCeRleveragesAgenticRAGsystemstocreatedescriptionfilesfromahigh-leveltextualdescriptionofthedesiredspecificationsandthenforauto-maticCRsdeploymentonremoteservers.Moreindetail,theprimarystrengthofARCeRliesinitsuser-friendlinessforinstructors,whocaneffortlesslycreateatrainingenvironmentbysimplyexpressingthedesiredcharacteristicsinnat-urallanguage,withoutthenecessityofconcernfortheunderlyingframework’sspecificsyntaxrequirements.ThisfeaturemakesCRmanagementaccessibletouserswithdifferentlevelsofexpertise.Itallowseffectiveusebyenablingad-vancedattackanddefenseconfigurations,adjustabledifficultylevels,andfullinfrastructurecustomizationtomeetspecifictrainingneedsinafullyautomatedmanner.

ARCeRdynamicallyandautomaticallyadaptstodifferentCRframeworksimplybychangingthesetofdocumentsprovidedasexternalknowledge.Thisensuresahighlevelofflexibilityinthatitremovesanydependencetospecificplatforms.Thismeansthatalterationsinthefield,suchastheintroductionofnewconfigurationpatternsorsupportfornovelscenarios,donotnecessitatemodificationstoARCeRlogic.Onlyarevisionofthereferencedocumentsisrequired,therebyfacilitatingmaintenanceandadaptationtoemergingrequire-ments.

Furthermore,byeliminatingtheneedtofine-tuneamodelforeachframe-work,ARCeRapproachdramaticallylowerscomputationalanddevelopmentcosts,makingCRgenerationmoreaffordable,fast,andadaptabletohetero-geneousscenarios.Finally,shouldmoreefficientLLMsemergeinthefuture,ARCeRcanbeupgradedwithouttheneedforacompletesystemrebuild.ThesecharacteristicsmaketheAgenticRAGperfectlysuitedtotheever-changingcon-textofCRs.

Thisworkdemonstrates(seeSection5)that,whilepureRAGsystemscanindeedaddressthetaskofgeneratingresponsesbasedonspecificdocumentationwithsufficienteffectiveness,theAgenticRAGapproachofARCeRleadstoenhancedperformance.

3.1ARCeRschema

TheoverallarchitectureandtheoperationofARCeRareillustratedinFig.2.TheAgenticRAGstructureofARCeRincludesanLLMthatoperatesasareasoningengineandtwoexternaltools:aRAGsubsystemandaCheckerTool.Finally,beingtheAIagentastatelesssystem,itisunabletorecallprevious

ARCER:anAgenticRAGforCyberRanges

interactionswiththeuser.Incontrast,ARCERhaseffectivememorymanage-ment,therebyconferringontheinstructortheflexibilitytomodifyCRfeaturesatasubsequenttime.

TheCheckerToolallowsARCERtocheckthesyntaxofthegeneratedoutputduringruntimeandtoperformself-correctionoferrorsiftheyoccur.

TheRAGsubsystemisinchargeofcompletingtheretrievalphaseofaRAGsystemusingtheMaximalMarginalRelevance[22](MMR)technique,whichaimstomitigateredundancyintheextractedchunksfromthevectorstore.Thisisachievedbyselectingthemostrelevantdocumentsforthequery,ensuringthosethatdifferthemostfromeachother,therebyprovidingtheAgenticRAGwithasubstantialamountofrelevantknowledge.Specifically,20chunksareinitiallyextracted,andafterfiltration,onlythe8mostrelevantonesaretransmittedtotheLLM.Toachieveanoptimalbalancebetweenrelevanceanddiversity,thespecificparameterlambda_multwassetto0.5[22].

Userinputmespage

LargeLangauge

Mode

syntax?

Cyberrangedescriptionfile

Agentoutput

syntaxchecker

Retrieved

documents

Userdocuments

VectorStore

Fig.2.Overallapproachschema

GivenARCER'ssupporttointeractwithdisparateCRframeworksitem-ploysdifferentvectorstorestoseparatelysavetheembeddings'chunksdoc-umentsrelatedtodifferentplatforms.Alldocuments(e.g.,frameworkusageguidesandexampleconfigurationfiles)arestoredinadatabasecontainingafolderwhosenamecoincideswiththenameofthespecificCRframework.TheRAGsubsystemexploitsformat-specificdocumentloaders(e.g.,forPDFandYAML)toimportthem.

Inmoredetail,thestepsperformedbyARCERarethefollowing:

(1)TheuserpromptisfedtotheLLMwhichparsesitandevaluateswhetheritisabletohandleitdirectlyorifitneedstocallanexternaltool.InthecaseitneedstoresorttotheRAGsubsystemitproducesanewspecificquerytobesenttotheRAGsubsystem.TheRAGsubsystemidentifiesthesuitablevectorstoreandthenitsearchesforrelevantchunksusingthe

8Lupinaccietal.

MMRtechnique.Onthebasisoftheadditionalknowledgecontainedinthesechunks,theLLMproducesaninitialpotentialoutputtotheuser’squery.

(2)TheoutputissenttoanexternalserverthatexposesanAPItotheCRplatformcomponentresponsibleforcheckingthesyntaxofthedescriptionfile.Theverificationstepisacriticalcomponentoftheoverallprocessasitensurestheintegrityofthegeneratedoutput.

(3)Iftheabovestepissuccessfullycompleted,theconfigurationfileiscreatedandsubsequentlyreturnedtotheuser.Intheeventthaterrorsaredetected,theyareautomaticallysentbacktoARCeRforself-correctionandthepro-cessstartsagainfromstep(1).

(4)Attheendofthedescriptionfilegenerationphase,ARCeRaskstheuserwhethertheywanttoproceedwiththeactualinstantiationoftheCR.Iftheuseragrees,itdirectlyexecutestheappropriatecommandsonthehostrunningtheCRframework.

3.2Implementationdetails

ThecurrentimplementationofARCeRusesAnthropic’sClaude3.7Sonnet[23]asLLMandSentenceTransformerModel[24]asembeddings.Fortheembeddingstorageanin-memoryvectorstoreisused.Thevectorstoreispopulatedwithchunksoftheexternal-supplieddocuments.Chunkshaveasizeof1000characterseachwithanoverlapof200characters.

ThechoiceoftheLLMwasmadebycomparingthreedifferentmodelsasdetailedinSection5.TheMMRmetricwasselectedbecauseityieldedbetterresultswithrespecttothecosinesimilaritymetric[25]initiallyusedwhichfre-quentlyreturnedthesamechunkmultipletimes,therebydiminishingthespecificknowledgeprovidedtotheLLM.

ARCeRhasbeenimplementedbyusingLangChain[18],aframeworkforthedevelopmentofapplicationspoweredbyLLMs,anditsextensionLang-Graph[26].LangChainimplementsastandardinterfacenotonlyforLLMsbutalsoforrelatedtechnologies,suchasembeddingmodelsandvectorstoresofhundredsofproviders.LangGraphisdesignedtobuildrobustandstatefulmulti-actorapplicationswithLLMsbymodelingstepsasedgesandnodesinagraph.LangChainhasbeenchosenbecauseofitspopularityinthescientificcommunityandbecauseofitspowerfulagent-creationlibraries.Beyondagentswithwell-definedhigh-levelinterfaces,LangGraphalsosupportscreatingagentsbackedbyalow-level,highlycontrollableAPI,enablingdeepcustomizationofagentlogic.

Byusingthepre-builtReActagentconstructorweimplementedtheretrievalandgenerationstepsoftheRAGsubsystemasacalltoexternaltools.Docu-mentchunks,retrievedfromthevectorstore,areincorporatedintothemessagesequencesenttotheLLMjustasiftheyweremessagesobtainedfrominvokinganyotherexternaltool.Thisapproachofferstheadvantagethatthemodelitselfgeneratesqueriesfortheretrievalphase,rewritingusermessagesintomoreef-fectivesearchqueries.Furthermore,itfullyautomatesdirectresponsesthatdonotrequirearetrievalphase,suchasresponsestogenericusergreetings.

ARCeR:anAgenticRAGforCyberRanges9

4Arealcasestudy:theCyRISframework

CyRIS(CyberRangeInstantiationSystem)[7]isanopen-sourcetoolthatpro-videsaflexible,scalable,andlow-costmechanismformanagingsecuritytrain-ingenvironment.CyRISsupportsKVMandAWSvirtualizationforseveralguestmachinesOS,includingUbuntu,CentOSandWindows.TheCyRISin-putfileisstructuredasaYAMLdocument,whichisdividedintothreesec-tions:host_settings,guest_settings,andclone_settings(seeListing1.1).ItshouldbenotedthatCyRISfacilitatesthedeploymentofCRinstancestooneormorehostservers.

Listing1.1.BasicCyRIScyberrangedescriptionfile

-host_settings:

id:host_1

mgmt_addr:localhost

virbr_addr:192.168.10.1

account:user

-guest_settings:

id:desktop

basevm_host:host_1

basevm_config_file:/home/user/images/basevm.xml

basevm_type:kvm

-clone_settings:

range_id:1

hosts:

-host_id:host_1

instance_number:1guests:

-guest_id:desktopnumber:1

entry_point:yes

topology:

-type:customnetworks:

-name:office

members:desktop.eth0

WechoseCyRISasCRinstantiationframeworkbecauseitisoneofthefewplatformsthatcanprovidesecurityfeaturesevenattheCRdescriptionstage,suchas:(i)trafficcapture,(ii)theabilitytoperformattacksandmalwareemu-lation,(iii)thepossibilitytoconfigurefirewall.Inaddition,theGitHubreposi-tory[27]isupdatedandmaintained,makingiteasytoinstallanduse.ThemainadvantageofCyRISisitsdetaileddocumentationnotonlyonthearchitectureoftheframework,butalsoonthesyntaxandsemanticstobeusedwhenwritingtheinputfile.Thisaspect,togetherwiththeexamplesofdescriptionfilesalreadyprovidedbytheauthors,isanexcellentknowledgebaseforARCeR.

Specifically,theinitialdocumentsprovidedtoARCeRasexternalknowl-edgeincludedthemostrecentversionofthepaperoutliningCyRIS,itsuserguide,andsixCRdescriptionfiles.Despitetheadequacyofthedocumentation,aninitialphaseofmanualdocumentanalysiswasnecessarytoremovepagescontaininginformationirrelevanttothepreparationofthedescriptionfile.ThepagesremovedincludedthosedescribingtheinstallationofCyRISandthosedocumentingperformanceanalysis.TomakeARCeRworkingwithCyRISweusedatotalof28pagesofdocumentation.

10Lupinaccietal.

NotwithstandingthepresentlimitationsofCyRISwithregardtothesup-portedsecurityfeatures,includingtheemulationofonlyafewtypesofattacksandmalware,CyRISemergedasanexcellentframeworkfortestingourapproach.

5Resultsanddiscussion

InthissectionweevaluatethecoveragethatARCeRoffersatthecurrentstagethroughdifferentmethodologicalstrategies.

First,acomparativeanalysiswasconductedamongtheoutputsgenerated:(i)byusingonlyanLLM,(ii)throughaRAGsystem,and(iii)byusingARCeR.Thecomparativeassessmentyieldedsignificantdiscrepanciesinaccuracy,com-pleteness,andcontextualrelevanceacrossthethreeapproaches.

Subsequently,weconductedbothqualitativeandquantitativeanalysesbygenerating20CRdescriptions,ofprogressivelyincreasingcomplexity,encom-passingallCyRIScharacteristics.ThistestenabledthemeasurementofthesuccessrateofARCeRacrossvaryingdifficultylevelsandconfigurationre-quirements,therebyprovidingrobustevidenceofthesystem’scapabilitiesunderdiverseoperationalscenarios.

Additionally,weevaluatedARCeR’sperformanceusingvarioustool-callingLLMsfromdifferentproviders:Claude3.7SonnetfromAnthropic,Gpt-4o-minifromOpenAI[28]andMistralLargefromMistralAI[29].Allmodelssuccessfullycompletedthetask,despitetheincreasingcomplexityoftheCRtobeinstan-tiated.However,minordiscrepanciesamongthemodelswereidentified,whichultimatelyguidedtheselectionofClaude3.7SonnetastheprimaryLLMforARCeR.

5.1LLM-SpecificconstraintsinCRgeneration

InordertoadequatelyinterfaceanLLMaskingforCRgeneration,somespecificconcernshavetobeproperlyaddressed.Inparticular,theamountofrequireddetailsintheuserpromptandthewayexternaltoolsareinvokedbotharebothspecifictotheemployedLLM.TheformerconcerncanbeeasilyresolvedbyspecifyingtheadditionalinformationneededbythespecificLLMinthesystemprompt,sothattheuserinputcanbekeptasgenericaspossible.Forexample,inthecaseofGpt-4o-mini,itwasnecessarytospecif