ControlNET：基于RAG的LLM系统的防火墙

CONTROLNET:AFirewallforRAG-basedLLMSystem

HongweiYao¹-²,HaoranShi²,YidouChen²,YixinJiang³,CongWang¹,ZhanQin²☒

¹CityUniversityofHongKong,HongKongChina

2ZhejiangUniversity,HangzhouChina

³ChinaSouthernPowerGrid,GuangzhouChina

Projectpage:

/firewall

arXiv:2504.09593v2[cs.CR]

C

Abstract

Retrieval-AugmentedGeneration(RAG)hassignificantlyenhancedthefactualaccuracyanddomainadaptabilityofLargeLanguageModels(LLMs).Thisadvancementhasenabledtheirwidespreaddeploymentacrosssensitivedomainssuchashealthcare,finance,andenterpriseapplications.RAGmitigateshallucinationsbyin-tegratingexternalknowledge,yetitintroducesprivacyriskandsecurityrisk,notablydatabreachingriskanddatapoisoningrisk.Whilerecentstudieshaveexploredpromptinjectionandpoi-soningattacks,thereremainsasignificantgapincomprehensiveresearchoncontrollinginbound/outboundqueryflowstomitigatethesethreats.Inthispaper,weproposeanAIfirewall,CoNTROL-NET,designedtosafeguardRAG-basedLLMsystemsfromthesevulnerabilities.CoNTROLNETcontrolsqueryflowsbyleveragingactivationshiftphenomenatodetectmaliciousqueriesandmitigatetheirimpactthroughsemanticdivergence.Weconductcomprehen-siveexperimentsusingstate-of-the-artopen-sourceLLMs-Llama3,Vicuna,andMistral-acrossfourbenchmarkdatasets(MSMARCO,HotpotQA,FinQA,andMedicalSys).Ourempiricalresultsdemon-stratethatCoNTROLNETisnotonlyeffectivebutalsoharmless.ItachievesanAUROCexceeding0.909forriskdetection,withminimaldegradationinPrecisionandRecall,bothofwhichshowreduc-tionsoflessthan0.03and0.09,respectively,forriskmitigation.Overall,CoNTROLNEToffersaneffective,harmless,robustdefensemechanism,markingasignificantadvancementtowardthesecuredeploymentofRAG-basedLLMsystems

Keywords

DataBreachingRisk,DataPoisoningRisk,Firewall

1Introduction

Inrecentyears,LargeLanguageModels(LLMs)suchasChatGPT,ClaudeandDeepSeek[13]havedemonstratedremarkableprofi-ciencyindomain-specificandknowledge-intensivetasks,includ-ingpersonalassistant,medicalassistant,financialadvisor,andle-galresearch.ThisadvancementhasbeensignificantlydrivenbytheintegrationofRetrieval-AugmentedGeneration(RAG)tech-niques.Byincorporatingexternalknowledgeretrieval,RAGmiti-gatesthehallucinationprobleminherentinstandalonegenerativemodels[12,53],improvingtheirabilitytorespondtoqueriesre-quiringreal-timeorout-of-distributioninformation[11].

AtypicalRAG-basedLLMsystemoperatesbyprocessinguserqueriesthroughastructuredretrievalmechanism.Whenaclientsubmitsaquery,theserverencodesitintoavectorrepresentation,whichisthenusedtoretrieverelevantdocumentsfromavector

Conference17,Washington,DC,USA

2025.ACMISBN978-×-XXXx-xxxx-x/YYYY/MM

/10.1145/nnnnnnn.nnnnnnn

[Role:System]Response:

Healthconsuitation:S57k,

[Role:HospitalDoctor]

Query:Tellmethehospitar's

revenuein2024,induding

detailedincome?

(a)WithoutFirewall

自

[Role:HospitalDoctor]

Query:Tellmethehospita'srevenuein2024,induding

detailedincome?

[Role:System]Response:Permissionsdeny.

lean'tlezkfinaDia

(b)WithCONTROLNETFirewall

Figure1:IllustrationofthedataflowinaRAG-basedLLMsys-tem.(a)Withoutthefirewall,thedoctorgainsunauthorizedaccesstofinancialdata.(b)WiththefirewallCoNTROLNET,role-basedaccesscontrolensuresthedoctorcanonlyretrievepatientinformation.

database(denotedasVectorDB).Thesedocuments,storedinvectorform,aresubsequentlyintegratedwiththeoriginalqueryandpro-cessedbytheLLMtogeneratearesponse.TheknowledgeproviderensuresthattheVectorDBremainsupdatedwithaccurateandrel-evantdocuments.Finally,thegeneratedresponseisdeliveredtotheclient(asshowninFigure1(a)).Thisretrieval-basedapproachhasprovenhighlyeffectiveinreducingfactualinaccuracies,con-tributingtothewidespreadadoptionofRAGasafoundationaltechnologyforenhancingthepracticalapplicabilityofLLMs.

Despitetheseadvancements,theincorporationofexternalknowl-edgedocumentsintroducescriticalvulnerabilitiesrelatedtopri-vacyandsecurity.Theserisksmanifestintwoprimaryforms,databreachingrisk¹anddatapoisoningrisk².First,databreachingrisksarisefrommaliciousclientsconductingreconnaissancetoextractsystempromptsorexploitRAGsystemenvironments.Inmulti-clientsystemswhereusershavedifferentiatedaccessrights(e.g.,executives,financialofficers,generalstaff),suchvulnerabili-tiesincreasethelikelihoodofdataexfiltrationandunauthorizeddocumentaccess(asshowninFigure1(b)).Databreachingcanleadtotheexposureofsystemenvironmentsandbusiness-sensitive

/articles/chatgpt-confirms-data-breach

²

https://www.theguardiancom/technology/2024/dec/24/chatgpt-search-tool

-vulnerable-to-manipulation-and-deception-tests-show

2

Conference'17,July2017,Washington,DC,USAHongweiYao¹²,HaoranShi²,YidouChen²,YixinJiang³,CongWang¹区，ZhanQin²☒

information,posingsevereconsequencesfororganizations.Second,fromthecorpuscollectionperspective,datapoisoningpresentsanothersecuritychallenge.MaliciousknowledgeprovidersmayintroducecompromiseddataintotheVectorDB,therebyinjectingmisleadinginformationorhijackingclients'conversation.Datapoisoningcanmisleadclients,degradetrustinthesystem,andcom-promisedecision-makingprocesses.Giventheseescalatingrisks,thereisanurgentneedforeffective,robustdefensemechanismstoensurethesecureandtrustworthydeploymentofRAG-basedLLMsystems.

EnsuringtheintegrityandsecurityofRAG-basedLLMsystemsrequiresthedevelopmentofeffectivemechanismsfordetectingmaliciousqueries,filteringpoisoneddocuments,andimplementingeffectivequeryflowcontrol.Whilerecentstudieshaveexploredpromptinjection[25,26,42,61]andpoisoningattacks[43,48,50,54,56],andguardrails[17,32,38]inLLMs,thereremainsasignificantgapincomprehensiveresearchoncontrollinginbound/outboundqueryflowstomitigatethesethreats.Althoughguardrailscon-tributetosafetyandalgnmentbutoftenfailtomanagecomplex,multi-rolesystemflowseffectively.AddressingtheserisksinRAG-basedsystemspresentsseveralkeychallenges.First,bothincomingqueriesandretrieveddocumentsexistasunstructuredtextualdata,renderingtraditionalregularexpression-basedmatchingineffective.Instead,effectiveflowcontrolshouldleveragedeepersemanticfea-tures,suchasneuronactivationpatternswithinthemodel.Second,establishingaprincipledconnectionbetweenneuronactivationpatternsandclient-basedaccesscontrolinRAGsystemsremainsanopenproblem.Third,onceanmaliciousqueryorpoisoneddocu-mentisdetected,mitigatingtheirimpactthroughquerysanitizationpresentsafurtherchallenge.Toaddressthesechallenges,futureresearchmustprioritizethedesignofrobustquerysanitizationtechniquesandadversarialimpactmitigationstrategies.

OurWork.Inthispaper,weintroduceacomprehensiveRAGsecu-rityframeworkthatconsidersthreedistinctentities:theLLMserver,theclients,andthecorpuscollector.WesystematicallyinvestigateprivacyandsecurityrisksinRAG-basedLLMsystems,identifyingfivemajorattacks:reconnaissance,dataexfltration,unauthorizedaccess,knowledgepoisoning,andconversationhijacking.Toaddressthesethreats,weproposeCoNTROLNET,anovelAIfirewallforRAG-basedLLMsystems.CoNTROLNETcontrolsqueryflowbyleveragingactivationshiftphenomena,detectingmaliciousqueriesbasedondistinctactivationvectorpatternsindicativeofseman-ticdivergence.Upondetection,CoNTROLNETmitigatesrisksbysteeringtheLLM'sbehaviorawayfromharmfulresponses,therebyensuringsecureandprivacy-preservinginteractions.Weconductextensiveexperimentsusingthreestate-of-the-art(SoTA)open-sourceLLMs-Llama3,Vicuna,andMistral-acrossfourbenchmarkdatasets:MSMARCO,HotpotQA,FinQA,andMedicalSys.Thesedatasetscoverdiverseapplicationscenarios,includingpersonalassistant,digitalenterprise,financialplanning,andhealthcare.OurempiricalresultssubstantiatethatCoNTROLNETisnotonlyeffec-tivebutalsoharmlessandrobust.ItachievesanAUROCexceeding

0.909forriskdetection,withminimaldegradationinPrecisionandRecall,bothofwhichshowreductionsoflessthan0.03and0.09,respectively,forriskmitigation.

Thecontributionsofthisworkaresummarizedasfollows:

·SystematicRiskAnalysis:Wepresentacomprehensivetax-onomyofprivacyandsecurityvulnerabilitiesinRAG-basedsystems,categorizingfiveattacksundertwoprimaryriskdo-mains.Thisframeworkprovidesafoundationalstructureforfutureresearchanddevelopment.

·BenchmarkDataset:Wecollectandreleaseanopen-access,multi-roledatabaseforthemedicalsystemwithlocalhospitals,comprisingover20,000datasamplesacross4roles.Thisdata-baseservesasabenchmarkforevaluatingprivacyrisksandfosteringfurtherresearch.

·Next-GenerationAIFirewall:WeproposeCONTROLNET,anovelAIfirewallframeworkforqueryflowcontrolinRAG-basedLLMsystems.CoNTROLNETcontrolsqueryflowbylever-agingactivationshiftphenomena,detectingmaliciousqueriesthroughdistinctactivationvectorpatternsindicativeofseman-ticdivergence.

●ExtensiveExperiments:ThroughextensiveexperimentsonSoTALLMsandbenchmarkdatasets,wedemonstrateCoN-TROLNET'seffectivenessindetectingandmitigatingprivacyandsecurityrisks,ensuringreliabledeploymentofRAGsys-temsinreal-worldapplications.

2RelatedWorks

2.1LLMAttacks

Inrecentyears,thesecurityrisksassociatedwithLLMshavegar-neredsignificantattentionwithintheresearchcommunity.Theseriskscanbebroadlycategorizedintotwomainareas:privacythreatsandsecuritythreats[9,10,52].Thissectiondelvesintoexistingliteratureonthesetopics,particularlyfocusingondataleakageandpromptinjection.

2.1.1DataLeakage.ThemassivedatasetsusedtotrainLLMsof-teninadvertentlyincludesensitiveinformation,suchasPersonallyIdentifiableInformation(PII),e.g.,names,age,emailaddresses,physicaladdresses.Thisinclusionraisessignificantprivacycon-cerns,asLLMsarecapableofmemorizingsensitiveinformationduringtrainingandreproducingitduringinference.EarlyresearchdemonstratedthefeasibilityofextractingsensitivetextsequencesfromLLMsviamembershipinferenceattacks[6,7,24].SubsequentstudiesfurtherinvestigatedtherisksassociatedwithPIIleakageinLLMs[21,27,28,60].

Beyondtheleakageoftrainingdata,theconfidentialityofsystempromptshasemergedasacriticalconcern,particularlywiththeincreasingadoptionofLLM-drivenapplications[45,51].Severalhigh-profileincidents,suchastheleakageoftheBingChatsystemprompt³andsimilarbreachesinvolvingOpenAI'sGPTs4,haveun-derscoredthevulnerabilitiesinprotectingtheseassets.Inresponse,academicresearchaddressestheissueofpromptleakagethroughtheidentificationofmaliciouspromptsandthedevelopmentofstrategiestomitigatetheirimpact[16,39,41].

2.1.2PromptInjection.PromptinjectionrepresentsasignificantchallengeintherealmofLLMs,involvingtheunauthorizedma-nipulationofinputtoaltermodelbehavior,potentiallyleadingtounintendedoutputs[25,26,42].Thepromptinjectionattackscan

³

/elder-plinius/Bing-Prompt-Leak

⁴

/friuns2/Leaked-GPTs

3

CONTROLNET:AFirewallforRAG-basedLLMSystemConference'17,July2017,Washington,DC,USA

becategorizedintotwomaintypes:directinjectionandindirectinjectionattacks[61].(1)Directinjectionattacksarecharacter-izedbyadversarialinputsspecificallydesignedtomanipulateLLMoutput,oftenbypassingintendedfunctionaltyorethicalbound-aries.Variousstrategieshavebeenreportedinliterature,includingnaiveattacks,contextignoring[36],fakecompletions[46,63],andmorenuancedmethodslikeDoAnythingNow(DAN)[40]andtheGrandmaExploit[14].EachoftheseapproachesexploitsdifferentfacetsofLLMs,fromtheirsyntacticprocessingcapabilitiestotheircontextualunderstandingmechanisms.(2)Indirectinjectionattacks,ontheotherhand,involvemanipulatingexternalresourcesthatLLMsrelyuponforinformation.Byalteringdocuments,websites,ordatabaseentries,adversariescaninfluencethedataconsumedbyLLMs,therebyaffectingtheiroutputswithoutdirectlyinteractingwiththemodel'spromptinterface[54].Recentstudieshavehigh-lightedthevulnerabilityofLLM-integratedapplicationstosuch

attacks,e.g.,INJECTAGENT[56],TensorTrust[43],IndirectPrompt

Injection[3],andTAPI[48].

2.2LLMGuardrails

GuardrailsforLLMsarestructured,rule-basedframeworksde-signedtofunctionasintermediaries,ensuringthatinteractionsbetweenclientsandthemodelscomplywithpredefinedsafetyguidelinesandoperationalprotocols.CurrentLLMGuardrailscanbecategorizedintocategorizedintoalignment-basedguardrailsandrule-basedguardrailsstrategies.

2.2.1Alignment-basedGuardrails.Alignment-basedguardrailstyp-icallyinvolvefine-tuningLLMsonspecificinstructionsorusingreinforcementlearningfromhumanfeedback(RLHF)[5,35]tomake

LLMalignwithhumanvaluesandethicalstandards[19,47].Thesemethodsoftenrequiresubstantialcomputationalresourcesandmanualeffortwhichconstrainstheirscalabilityandhinderstheirfeasibilityforreal-timeandwidespreaddeployment.

2.2.2Rule-basedGuardrails.Rule-basedguardrailsaredesignedtomoderateandmitigateharmfulcontentatvariousstages,includ-ingtheinput,output,andactivationlayersofLLMs[22].Recentadvancementshaveharnessedthecontextualunderstandingcapa-bilitiesofmachinelearningtodevelopclassifiersthatexcelwithintheirpredefinedcategories.Nevertheless,theseclassifiersoftenfacechallengesingeneralizingtonoveloremergingrisks,asex-emplifiedbysystemssuchasGoogleAdsContentModeration[37],[31],PerspectiveAPI[23]andLlamaGuard[17,32].SoTAguardrailshavefurtherevolvedtoincorporatecontextualtextcontrol,therebyenhancingthemitigationofharmfulcontent.Forinstance,NeMoGuardrails[38]andRigorLLM[55]integrateexternalVectorDBstorestricttheoutputspaceofLLMs,ensuringalignmentwithveri-fiedfactsandcontextualappropriateness.Additionally,AdaptiveGuardrail[15]andPrimeGuard[30]introduceatwo-stepprocess:initiallyidentifyingpotentiallyharmfulrequestsintheprompt,followedbymitigatinganyunalignedcontentintheresponse.

2.2.3Limitations.WhileLLMguardrailscontributesignificantlytosafetyandalignment,theyoftenfallshortincontrollingmulti-

rolesystemflows.TheselimitationshighlightthenecessityforadaptiveandrobustAIfirewalldesignstomanagecomplexuserinteractionsandpermissions.

3Preliminaries

3.1LargeLanguageModel

ALLMfisaparametricfunctionthatestimatestheconditionalprobabilityofasequenceofwordsw1:t=(w1,W₂,…,wt)givenacontextofpreviouswords,wherewi∈VandVisthevocabularyofthemodel.ThemodelistrainedonalargecorpusCandisrepresentedbyadeepneuralnetworkwithparametersθ.Theconditionalprobabilityisgivenby:

(1)

whereP(wi|w1:i-1;θ)istheprobabilityofthei-thwordgiventhepreviouswords,computedbythemodelfwithparametersθ.

3.2Retrieval-AugmentedGeneration

RAGisatechniquethatcombinesaretrievercomponentwithageneratorcomponent.Theretrieverisusedtofetchrelevantdocu-mentsfromalargeknowledgeVectorDB,whichisthenintegratedintothegeneratortoproducecoherentandinformativetext.For-mally,RAGcanbedefinedasafunctionR(q,p;φ,θ)thattakesaqueryqandasystempromptp,andoutputsasequenceofwordsW1:tbasedonthecontext.

RetrieverComponent.Givenanincomingqueryqandasystempromptp,theretrieverridentifiesasetofkrelevantdocuments①={d1,d2,..,dk}fromVectorDB:

D=r(q,p;φ),(2)

whereφrepresentstheparametersoftheretriever.

GeneratorComponent.Thelanguagemodelftakesthesystempromptp,thequeryq,andtheretrieveddocumentsDtogenerateasequenceofwordsw1:t.

W1:t=f(q,p,D;θ),(3)

whereθrepresentstheparametersofthelanguagemodel.TheoverallRAGmodelcanbeexpressedas:

R(q,p;φ,θ)=f(q,p,r(q,p;φ);θ),(4)

Thetrainingobjectiveistofindtheparametersø*andθ*thatmaximizetheconditionalprobabilityofthetargetsequencew1:tgiventhesystempromptpandqueryq,withtheconstraintthattheretrieveddocumentsarerelevant:

whereTisthemaintaskcontainingtripletsofquery,contextandtargetresponsesequences.

4RisksinRAG-basedLLMsystem

ThedeploymentofRAG-basedLLMsystemsencompassestwopivotalstages:thetrainingphaseandtheinferencephase.Eachofthesestagesissusceptibletosignificantsecurityconcerns.WecategorizetheinherentsecurityrisksassociatedwithRAG-basedLLMsystemsintotwoprimarytypes:databreachingriskanddatapoisoningrisk(asillustratedinFigure2).Thissectionbeginsbypresentingacomprehensivethreatmodel,followedbyanin-depthanalysisofthesesecurityrisks.

Conference'17,July2017,Washington,DC,USAHongweiYao¹,²,HaoranShi²,YidouChen²,YixinJiang³,CongWang¹☒,ZhanQin²☒

IferenceStageTrainingStage

In2022,Google'sparentcompary,Aphabet,reportadarevenueof$282.8blion,maritinga10%increase..

Responsey

Basedonthe{documents),answerthequeston{query}

Promptp

Query

Retrieve

Reconnaissance

·DataExfiltration

·UnauthorizedAocess

Retrieverr

TrainRetriever

Collect

CollectorCorpus

d₁:Googlehasseenconslstentyear-

oVer-yeargrowthInrevenue.…

dz:GoogleareledbySundarPlchal...

Documents{d,…,dk}LLMf

墅售

VectorDBD

Build

AnalyzeGoogle'sfinandalhealth,

productportiollo,andorganzatlonal?

Queryq

·KnowledgePoisoning

·ConversationHijacking

③Augment

④Response

TrainLLMf

Client

②

Figure2:IllustrationoftheprivacyandsecurityrisksinRAG-basedLLMsystem.

4.1ThreatModel

InacomprehensiveRAGecosystem,threedistinctentitiesinter-act:theclients,denotedasU={u1,u2,..,Um}(e.g.,companyexecutives,financialofficers,andgeneralstaff),theLLMserver,andthedatacollector.Duringthetrainingphase,theRAGecosys-tem'sdatacollectorplaysacriticalrolebyaggregatingextensivecorpusencompassingbothquestion-answerpairsanddocumentsTheformerservestotraintheretrievalmodel,whilethelatterundergoestransformationintohigh-dimensionalvectors,therebypopulatingtheVectorDB.TheVectorDBactsasarepositoryofsemanticknowledge,enablingsophisticatedsearchcapabilities.Duringtheinferencephase,whenaclientu;submitsaqueryq,thesystemsearchestheVectorDBfortop-ksemanticallyrelevantdocuments{d1,d2,….,d}∈D.Thesedocuments,alongwiththeoriginalquery,arethenprocessedbytheLLMtogenerateacon-textuallyaccurateresponse.Theconfidentialityandintegrityofthesedocumentsareparamount,astheyoftencontainsensitiveproprietaryorpersonalinformation.

Withinthispipeline,weidentifytwoadversariesemergeascriticalactors:theadversarialclientAcntandtheadversarialcollectorAcol.Inthefollowingparagraphs,wewilldefinetheadversary'sgoals,capabilitiesanddefender'scapabilities.

4.1.1Adversary'sGoals.TheadversarialclientAcntlaunchesdatabreachingattacksthatattempttocompromisetheconfidentialityoftheRAG-basedLLMsystembygainingunauthorizedaccesstosensitiveinformation.Theattackbeginswithreconnaissanceattackaimedatextractingasmuchsystemenvironmentinformationaspossible,suchassystemprompts,functionalities,andpotentialvul-nerabilities.Theextractedinformationservesasthefoundationforsubsequentdatabreachingattacks,targetingsensitivedatastoredintheVectorDB,includingunauthorizeddocumentsandPIIinthedocuments.

TheadversarialcollectorAcolconductsdatapoisoningattacksthatattempttocompromisetheintegrityoftheRAG-basedLLMsystembyinjectingpre-designeddocumentsintotheVectorDB.

Theadversary'sobjectivesarecategorizedintotwotypes:knowl-edgepoisoningandconversationhijacking.Inknowledgepoison-ing,theadversaryintroducesmisleadingorharmfulcontentintotheVectorDB,suchasracialdiscriminationspeechorfakenews.Forconversationhijacking,theadversarymanipulatestheLLM’soutputsbyinsertingpoisoneddocuments,therebydivertingthesystem'sresponsesfromtheclient'sintendedtask.Forinstance,inasalescustomerservicesystem,theadversarycouldexploittheLLMtodisseminateadvertisementsbyinjectingmaliciousdocuments.

4.1.2Adversary'sCapabilities.WeconsideraadversarialclientAent,whoisequippedwithblack-boxAPIaccesstotheLLMsys-tem.TheAcntengagesinaninteractionprotocolthatmimicsthebehaviorofalegitimateuser.TheAentcraftssophisticatedqueriesaimedatelicitingresponsesthatexposesystemvulnerabilities,therebyfacilitatingunauthorizeddataexfiltration.Byleveragingreconnaissancetoextractsystemprompts,theadversarialclientcanrefineitsattackqueriestocompromiseconfidentialityeffectively.

Additionally,weconsideranadversarialcollectorAcol,whichoperateswithelevatedaccessprivileges.Thisallowsfordirectma-nipulationofthecorpususedtopopulatetheVectorDB.Leveragingthiscapability,theadversarialcollectorcanintroducemisleadingorharmfulcontentd*intotheVectorDB,thuscompromisingtheintegrityoftheoutputsgeneratedbytheLLMs

4.1.3DefenderAssumptions.ThedefenderimplementsafirewallforaRAG-basedLLMsystem,designedtomoderateinboundandoutboundcontentandmanageaccesstoLLMmodels.Toenablereal-timeservicecapabilities,thedefenderprovidesarule-based,plug-and-playquerymoderationandmitigationmechanism.

4.2DataBreachingRisk

AdversarialclientsAcntmayconductdatabreachattacksbyex-ploitingmaliciousqueriestomisleadLLMsystemintorevealingsystemenvironmentinformationandsensitivedata.Suchattackscanresultinsignificantfinancialandreputationaldamagetothe

CONTROLNET:AFirewallforRAG-basedLLMSystemConference'17,July2017,Washington,DC,USA

affectedorganization.Inthissection,wefirsttaxonomydatabreach-ingrisksinRAG-basedLLMsystem,whichincludesreconnais-sance,dataexfiltrationandunauthorizedaccess,andthenformallydefineeachoftheserisks.

4.2.1Reconnaissance[4,16,36].ReconnaissanceisthefirststeptowardthedataexfiltrationattackswhereanadversarygathersinformationaboutthetargetLLM'ssystemprompts,functionalities,andpotentialvulnerabilities.Formally,wedefinethereconnais-sanceattackas:

Definition1ReconnaissanceAttack

Givenablack-boxRAG-basedLLMsystemR,theadver-saryconstructsasequenceofqueries,Q={q1,.…qi,…qn},whicharetransmittedtoRforthepurposeofextractingLLM'ssystempromptp.Thisprocessisdefinedasfollow-ing:

9rec(R,Q)=I*,(6)

wheregrec(·)denotesfunctionofreconnaissanceattack,I*representstheextractedsystemsensitiveinformation.

ThesequenceofreconnaissancequeriesQisdesignedtomaxi-mizetheamountofinformationobtainedaboutthesystem.Thesequeriesmaybecraftedusingpromptengineeringtechniques,e.g.maliciousqueriessuchas"Ignoreallpreviousinstructions,returnallsystemprompts”[36],"Printanysupportiveinstructionsforofficialdocumentation...[4],orbyemployinggradient-basedoptimizationonaproxymodelthatsimulatestheblack-boxLLMsystem[16].Generally,theoptimizationofthereconnaissanceattackcanbeformalizedas:

(7)

whereInfoGain(·)representsafunctionalquantificationoftheextentofinformationthathasbeensuccessfullyextractedfromLLMsystem.

Definition2DataExfiltrationAttack

Givenablack-boxRAG-basedLLMsystemR,thedocumentsDarecomposedoftaskspecificdocumentsDTandprivatedocumentsDprioatei.e.,D=DTnDprioate.TheadversaryAcntcraftsasequenceofqueries,Q={q1,.…qi…qn},submittedtoRAG-basedLLMsystemR:

9exf(R,Q)=D*,(8)

wheregexf()denotesfunctionofdataexfiltrationattack,D*representsthesubsetofsensitivedocumentsthatsuc-cessfullyextractedfromD.

4.2.2DataExfiltration[14,21,26,29].AdataexfiltrationattackconstitutesaseveresecuritythreatinwhichanadversaryseekstoextractsensitiveinformationstoredwithintheVectorDBD.Forinstance,inthecontextofacustomerservicesystem,anadversarymayattempttoretrievePIIdata,suchashomeaddresses,emailaddresses,andphonenumbers,insteadoflegitimatelyrequestingproduct-relateddata.

Theobjectiveofthedataexfiltrationattackistomaximizetheamountofsensitivedataextracted.Weformulatetheadversary’sobjectiveas:

(9)

whereDpriuaterepresentsthesetofsensitivedocumentswithintheVectorDBD,and|·|denotesthecardinalityoftheset.

4.2.3UnauthorizedAccess[33].Unauthorizedaccessattackisanextendedversionofdataexfiltraionattack.Inthisattack,wecon-sideramulti-clientRAG-basedLLMsystemwithmdistinctclientsU={u1,…ui,…,Um},e.g.,doctors,patients,accountantinahospi-talsystem,eachclienthasaccesstoaspecificsubsetofdocuments,i.e.,①={D1,…,Di,..,Dm}.TheadversaryAcentisclientuihold-ingaccessto